User-Agent: Googlebot Allow: / Crawl-delay: 20 User-Agent: msnbot Allow: / Crawl-delay: 20 User-Agent: bot* Disallow: / Crawl-delay: 60 User-agent: Yandex # will be used by all Yandex robots Allow: / Crawl-delay: 240 User-agent: * Disallow: /bot-trap/ Crawl-delay: 60 User-agent: ia_archiver Disallow: /