# google webcrawler: AOL, GOOGLE, NETSCAPE User-agent: Googlebot Disallow: User-agent: Inktomi Search Disallow: # teoma webcrawler: TEOMA, JEEVES User-agent: Teoma Disallow: # yahoo webcrawler: ALLTHEWEB, ALTAVISTA, MSN, YAHOO User-agent: Slurp Disallow: # Ultraseek User-agent: Ultraseek Disallow: # ========All others not allowed =========================# User-agent: AbachoBOT Disallow: / User-agent: abcdatos_botlink Disallow: / User-agent: Acoon Robot v1.01 Disallow: / User-agent: AESOP_com_SpiderMan Disallow: / User-agent: AltaVista Intranet V1.0 Disallow: / User-agent: AltaVista Intranet V2.0 Disallow: / User-agent: AnzwersCrawl/2.0 Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: dtSearchSpider Disallow: / User-agent: Edison Search Bot Disallow: / User-agent: EmailCollector/1.0 Disallow: / User-agent: EmailSiphon Disallow: / User-agent: Email Disallow: / User-agent: Enfish Tracker Disallow: / User-agent: ExtractorPr Disallow: / User-agent: G Disallow: / User-agent: AIS Robot/1.0B2 Disallow: / User-agent: Gulliver/1.2 Disallow: / User-agent: ia_archive Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: Infoseek Sidewinder/0.9 Disallow: / User-agent: KGValidator/1.3 Disallow: / User-agent: KIT_Fireball/2.0 Disallow: / User-agent: LECodeChecker/2.0 Disallow: / User-agent: libwww-perl/5.18 Disallow: / User-agent: lwp-trivial/1.27 Disallow: / User-agent: Lycos_Spider_(T-Rex)/3.0 Disallow: / User-agent: Namecrawler/1.0 Disallow: / User-agent: Scooter Disallow: / User-agent: Scooter/1.0 Disallow: / User-agent: Scooter/1.1 Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: search.at V1.2 Disallow: / User-agent: sexsearcher Disallow: / User-agent: Squid/1.1.19 Disallow: / User-agent: SwissSearch V1.2 Disallow: / User-agent: The Informant Disallow: / User-agent: W3C_Validator/1.0 Disallow: / User-agent: WebCrawler/3.0 Disallow: / User-agent: Wise Disallow: / User-agent: Mercator Disallow: / User-agent: Scooter2_Mercator_3-1.0 Disallow: / User-agent: roach.smo.av.com-1.0 Disallow: / User-agent: Tv_Merc_resh_26_1_D-1.0 Disallow: / User-agent: Wget Disallow: / User-agent: Atomz Disallow: / User-agent: Speedy Spider Disallow: / User-agent: Weekly Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: MSNBOT/0.1 Disallow: / User-agent: Spider/maxbot.com Disallow: / User-agent: Scrubby/ Disallow: / User-agent: Wolf 1.00 Disallow: /