#bad bots User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot) User-agent: MSIE 6.0 User-agent: Java/1.5.0_* ## please use a more specific useragent User-agent: Java/1.5.0_11 ## Disallow: / #everyone else User-agent: * ## Crawl-delay: 100 Disallow: /cgi-bin/ Disallow: /comments/flag/ Disallow: /mailfriend/ Disallow: /events/ Disallow: /r/ ## Bots shouldn't go into this URL