Возраст домена | n/a |
Дата окончания | n/a |
PR | 3 |
ИКС | |
Страниц в Google | 18200 |
Страниц в Яндексе | 257 |
Dmoz | ![]() |
Яндекс Каталог | Нет |
Alexa Traffic Rank | 3512517 |
Alexa Country | Нет данных |
История изменения показателей | Авторизация |
Идет сбор информации... Обновить
Pferde kaufen, Pferde verkaufen und Pferde suchen - Westernpferde kaufen - HorseBase Pferdemarkt
Pferde kaufen, Pferde verkaufen, Pferde, Pferdemarkt, Pferdeangebote, Pferdeanzeigen, Pferdebörse, Pferdeboerse, Pferdeinserate, Pferdeverkauf, Pferdekauf, Pferde inserieren, Pferde kaufen, Pferdehandel, Pferd kaufen, Pferdekauf, Deckhengste, Deckanzeigen
Pferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.
UTF-8
61.54 КБ
1 327
10 807 симв.
9 323 симв.
Данные предоставлены сервисом semrush
Идет сбор информации... Обновить
Счетчик | Посетители за 24 часа | Просмотры | Просмотров на посетителя |
---|---|---|---|
![]() | Нет доступа | Нет доступа | n/a |
Domain: horsebase.de
Nserver: ns.second-ns.com
Nserver: ns1.your-server.de
Status: connect
Changed: 2016-12-13T22:25:36+01:00
[Tech-C]
Type: PERSON
Name: Martin Hetzner
Organisation: Hetzner Online GmbH
Address: Industriestr. 25
Address: DE
PostalCode: 91710
City: Gunzenhausen
CountryCode: DE
Phone: +4998315050
Fax: +4998315053
Email: info@hetzner.de
Changed: 2015-07-29T16:32:44+02:00
[Zone-C]
Type: PERSON
Name: Martin Hetzner
Organisation: Hetzner Online GmbH
Address: Industriestr. 25
Address: DE
PostalCode: 91710
City: Gunzenhausen
CountryCode: DE
Phone: +4998315050
Fax: +4998315053
Email: info@hetzner.de
Changed: 2015-07-29T16:32:44+02:00
User-agent: WebReaper
User-agent: AlphaSeoBot
User-agent: AlphaSeoBot-SA
User-agent: WebCopier
User-agent: Offline Explorer
User-agent: HTTrack
User-agent: Microsoft.URL.Control
User-agent: EmailCollector
User-agent: penthesilea
User-agent: MJ12bot
User-Agent: Topsy
User-agent: spbot
User-agent: dotbot
User-agent: SeznamBot
Disallow: /
User-agent: OrangeBot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-Agent: Linkdexbot/2.0
Disallow: /
User-Agent: Linkdexbot/2.1
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-agent: Sogou web spider Sogou web spider/4.0
Disallow: /
User-agent: sogou web spider
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: Opoint
Disallow: /
User-agent: 008
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: ClarityDailyBot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; ClarityDailyBot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
Disallow: /
User-agent: PAD-bot/9.0
Disallow: /
User-agent: PAD-bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)
Disallow: /
User-agent: SoftListBot/2.2
Disallow: /
User-agent: SoftListBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: NativeHost
Disallow: /
User-agent: newsme/1.0; feedback@news.me
Disallow: /
User-agent: newsme/1.0
Disallow: /
User-agent: newsme
Disallow: /
User-agent: Niki-Bot
Disallow: /
User-agent: NING/1.0
Disallow: /
User-agent: NING
Disallow: /
User-agent: peerindex
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Disallow: /
User-agent: Diffbot/0.1
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net)
Disallow: /
User-agent: elefent
Disallow: /
User-agent: elefent/1.2
Disallow: /
User-agent: crawler.elefent.net
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 4.0
Disallow: /
User-agent: MS Search 5.0
Disallow: /
User-agent: MS Search 6.0
Disallow: /
User-agent: MS Search 7.0
Disallow: /
User-agent: MS Search
Disallow: /
User-agent: CCBot/2.0 (http://commoncrawl.org/faq/)
Disallow: /
User-agent: CCBot/2.0
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
Disallow: /
User-agent: Mail.RU_Bot/Robots/2.0
Disallow: /
User-agent: AOL_Daniel_Clark_Spider/Nutch-0.9 (AOL Search; danielaclark1@aol.com)
Disallow: /
User-agent: AolOnScraper/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html; gzip)
Disallow: /
User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /
User-agent: Apache Sites Search Facet
Disallow: /
User-agent: bigdata
Disallow: /
User-agent: Fast corporate crawler
Disallow: /
User-agent: 008
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: magpie-crawler/1.1
Disallow: /
User-agent: MagpieRSS
Disallow: /
User-agent: psnsearch
Disallow: /
User-agent: SLCC2
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: GuzzleHttp
Disallow: /
User-agent: Go 1.1 package http
Disallow: /
User-agent: Go 1.1
Disallow: /
User-agent: Go 1.2
Disallow: /
User-agent: Go 1.2 package http
Disallow: /
User-agent: gocrawl v0.4
Disallow: /
User-agent: gocrawl
Disallow: /
User-agent: Google-HTTP-Java-Client
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: Hatena Star
Disallow: /
User-agent: PagePeeker.com
Disallow: /
User-agent: PagePeeker
Disallow: /
User-agent: _agentname
Disallow: /
User-agent: agentname
Disallow: /
User-agent: 1 in Business Networking, www.MarcusWalker.ws
Disallow: /
User-agent: Business Networking
Disallow: /
User-agent: AntiOpensourceBot/1.1
Disallow: /
User-agent: AntiOpensourceBot
Disallow: /
User-agent: antibot-V1.1.5/i586-linux-2.2
Disallow: /
User-agent: Anti-XBOCT/2007 [xxx] (OS; X; JU)
Disallow: /
User-agent: AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Disallow: /
User-agent: Anti-XBOCT
Disallow: /
User-agent: AntBot
Disallow: /
User-agent: Ant-Nutch
Disallow: /
User-agent: AntBot/1.0
Disallow: /
User-agent: AnzwersCrawl/2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: Wotbox/2.01 (+http://www.wotbox.com/bot/)
Disallow: /
User-agent: Wotbox
Disallow: /
User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPicker /1.0
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-Agent: litefinder
Disallow: /
User-agent: Friendica
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Flamingo
Disallow: /
User-agent: Faraday v0.8.8
Disallow: /
User-agent: Faraday
Disallow: /
User-agent: Barkrowler 0.1.6
Disallow: /
User-agent: Barkrowler
Disallow: /
User-Agent: ExperiBot_v1
Disallow: /
User-Agent: ExperiBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-Agent: omgilibot
Disallow: /
User-Agent: omgili
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: hloader
Disallow: /
User-agent: ConveraCrawler/0.9e
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: YahooFeedSeeker/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://publisher.yahoo.com/rssguide)
Disallow: /
User-agent: YahooFeedSeeker/2.0+
Disallow: /
User-agent: YahooFeedSeeker
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Disallow: /
User-agent: ExtLinksBot/1.5
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: NCBot
Disallow: /
User-agent: Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html)
Disallow: /
User-agent: Testbot/1.0
Disallow: /
User-agent: Testbot
Disallow: /
User-agent: Bot !Susie
Disallow: /
User-agent: Slurp Disallow: /
User-agent: !Susie
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Disallow: /
User-agent: GroupHigh/1.0
Disallow: /
User-agent: G-i-g-a-b-o-t
Disallow: /
User-agent: SeznamBot/3.2
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Disallow: /
User-agent: Helloo/1.0.4
Disallow: /
User-agent: Mozilla/5.0 (compatible; Helloo/1.0.4; +http://www.helloo.io/;)
Disallow: /
User-agent: SiteExplorer/1.1b
Disallow: /
User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /
User-agent: Mail.RU_Bot/Robots
Disallow: /
User-agent: attach
Disallow: /
User-agent: ASPSeek
Disallow: /
User-agent: appie
Disallow: /
User-agent: AbachoBOT
Disallow: /
User-agent: autoemailspider
Disallow: /
User-agent: anarchie
Disallow: /
User-agent: antibot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: B2w
Disallow: /
User-agent: BackWeb
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: Bandit
Disallow: /
User-agent: BatchFTP
Disallow: /
User-agent: Black\ Hole
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bot\ mailto
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: Buddy
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: bumblebee
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: ClariaBot
Disallow: /
User-agent: curl
Disallow: /
User-agent: clsHTTP
Disallow: /
User-agent: ChinaClaw
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: Collector
Disallow: /
User-agent: COAST\ WebMaster
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: ColdFusion
Disallow: /
User-agent: Copier
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DA
Disallow: /
User-agent: DTS\ Agent
Disallow: /
User-agent: DISCo\ Pump
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Diamond
Disallow: /
User-agent: Download\ Demon
Disallow: /
User-agent: Download\ Wonder
Disallow: /
User-agent: Downloader
Disallow: /
User-agent: dloader
Disallow: /
User-agent: Drip
Disallow: /
User-agent: eCatch
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: Express\ WebPictures
Disallow: /
User-agent: Extreme\ Picture\ Finder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EasyDL
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: EyeNetIE
Disallow: /
User-agent: FAST\ WebCrawler
Disallow: /
User-agent: masscan/1.0 (https://github.com/robertdavidgraham/masscan)
Disallow: /
User-agent: masscan/1.0
Disallow: /
User-agent: masscan
Disallow: /
User-agent: semantic-visions.com crawler
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: ADmantX Platform Semantic Analyzer
Disallow: /
User-agent: proximic
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: GetintentCrawler
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: zgrab
Disallow: /
User-agent: Pigafetta
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: postano
Disallow: /
User-agent: Prlog
Disallow: /
User-agent: PulseCrawler/1.1
Disallow: /
User-agent: QuerySeekerSpider
Disallow: /
User-agent: quipu
Disallow: /
User-agent: Raven
Disallow: /
User-agent: RavenCrawler
Disallow: /
User-agent: RBot
Disallow: /
User-agent: Recorded Future
Disallow: /
User-agent: Riddler
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: RukiCrawler
Disallow: /
User-agent: SafeDNSBot
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Searchie
Disallow: /
User-agent: semantic-visions.com crawler
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: seocharger-robot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SEOlyticsCrawler
Disallow: /
User-agent: seoscanners
Disallow: /
User-agent: seoscanners.net
Disallow: /
User-agent: ShowyouBot (http://showyou.com/crawler)
Disallow: /
User-agent: simplecrawler
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-Agent: siteluxbot
Disallow: /
User-agent: SmeshBot
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: srmse/Nutch
Disallow: /
User-agent: SoftListBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: squirrly
Disallow: /
User-agent: ssearch_bot
Disallow: /
User-agent: Stratagems Kumo
Disallow: /
User-agent: tbot-nutch
Disallow: /
User-agent: TWMBot
Disallow: /
User-agent: Typhoeus - https://github.com/typhoeus/typhoeus
Disallow: /
User-agent: uipbot
Disallow: /
User-agent: uMBot-LN
Disallow: /
User-agent: UptimeRobot/2.0
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: WebIndex
Disallow: /
User-agent: VeBot
Disallow: /
User-agent: VeriCiteCrawler
Disallow: /
User-agent: WeSEE_Bot
Disallow: /
User-agent: WiseGuys Robot
Disallow: /
User-agent: voltron
Disallow: /
User-agent: woobot
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: X-CAD-SE
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: Yahoo Pipes 1.0
Disallow: /
User-agent: YioopBot
Disallow: /
User-agent: crawler.vs.uni-duisburg-essen.de
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: FileHound
Disallow: /
User-agent: Fetch\ API\ Request
Disallow: /
User-agent: FlashGet
Disallow: /
User-agent: FlickBot
Disallow: /
User-agent: FrontPage
Disallow: /
User-agent: FreeFind.com
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: GetSmart
Disallow: /
User-agent: Generic
Disallow: /
User-agent: Go!Zilla
Disallow: /
User-agent: Go-Ahead-Got-It
Disallow: /
User-agent: gotit
Disallow: /
User-agent: Grabber
Disallow: /
User-agent: GrabNet
Disallow: /
User-agent: Grafula
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: HMView
Disallow: /
User-agent: Heretrix
Disallow: /
User-agent: HitboxDoctor
Disallow: /
User-agent: HTTPapp
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTPTrack
Disallow: /
User-agent: HTTPviewer
Disallow: /
User-agent: httplib
Disallow: /
User-agent: httpfetcher
Disallow: /
User-agent: httpscraper
Disallow: /
User-agent: hloader
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: InterGET
Disallow: /
User-agent: Internet\ Ninja
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: InternetSeer.com
Disallow: /
User-agent: Iria
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: JetCar
Disallow: /
User-agent: JOC
Disallow: /
User-agent: JOC\ Web\ Spider
Disallow: /
User-agent: JoBo
Disallow: /
User-agent: Java
Disallow: /
User-agent: JustView
Disallow: /
User-agent: Jonzilla
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin\ Spider
Disallow: /
User-agent: Keyword\ Density
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LeechFTP
Disallow: /
User-agent: Lachesis
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb
Disallow: /
User-agent: Libby_
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: lftp
Disallow: /
User-agent: likse
Disallow: /
User-agent: Link
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp\ request
Disallow: /
User-agent: Magnet
Disallow: /
User-agent: Mag-Net
Disallow: /
User-agent: Mass\ Downloader
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Missigua
Disallow: /
User-agent: Mewsoft\ Search\ Engine
Disallow: /
User-agent: moget
Disallow: /
User-agent: Mata\ Hari
Disallow: /
User-agent: Memo
Disallow: /
User-agent: Metacarta
Disallow: /
User-agent: Mercator
Disallow: /
User-agent: MIDown\ tool
Disallow: /
User-agent: MFC_Tear_Sample
Disallow: /
User-agent: Mirror
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Mister\ PiX
Disallow: /
User-agent: NationalDirectory\ WebSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Nikto
Disallow: /
User-agent: Navroad
Disallow: /
User-agent: NearSite
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NetResearchServer
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: Net\ Vampire
Disallow: /
User-agent: Net\ Probe
Disallow: /
User-agent: NetZip
Disallow: /
User-agent: nexuscache
Disallow: /
User-agent: Ninja
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: our\ agent
Disallow: /
User-agent: onestop
Disallow: /
User-agent: oBot
Disallow: /
User-agent: Octopus
Disallow: /
User-agent: Offline\ Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind\ data\ gatherer
Disallow: /
User-agent: PageGrabber
Disallow: /
User-agent: Papa\ Foto
Disallow: /
User-agent: Xaldon\ WebSpider
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: Xara
Disallow: /
User-agent: Y!TunnelPro
Disallow: /
User-agent: YahooYSMcm
Disallow: /
User-agent: Zade
Disallow: /
User-agent: ZBot
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Rogerbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Dotbot
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: Xaldon\ WebSpider
Disallow: /
User-agent: Screaming Frog SEO Spider/8.1
Disallow: /
User-agent: DoBot
Disallow: /
User-agent: Laserlikebot
Disallow: /
User-agent: oBot
Disallow: /
User-agent: Neliveto
Disallow: /
User-Agent: ahrefs
Disallow: /
User-Agent: AHC/1.0
Disallow: /
User-Agent: AHC
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Disallow: /
User-agent: evc-batch/2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Disallow: /
User-agent: SemrushBot/1.2~bl
Disallow: /
User-agent: evc-batch
Disallow: /
User-agent: 007ac9 Crawler
disallow: /
User-agent: 1 2 3 Submit PRO
disallow: /
User-agent: 200PleaseBot
disallow: /
User-agent: 2ADAMbot
disallow: /
User-agent: 2ADAMbot/1.0
disallow: /
User-agent: 360Spider
disallow: /
user-agent: Abonti
disallow: /
user-agent: Abonti/0.92
disallow: /
user-agent: abot v1.0
disallow: /
user-agent: aboutthedomain
disallow: /
user-agent: Add Catalog
disallow: /
user-agent: Add Catalog/2.1
disallow: /
user-agent: AdvBot
disallow: /
user-agent: AdvBot/2.0
disallow: /
user-agent: AhrefsBot
disallow: /
user-agent: Ahrefs-Bot
disallow: /
user-agent: AhrefsBot/1.0
disallow: /
user-agent: Ahrefs-Bot/1.0
disallow: /
user-agent: Ahrefs-Bot/2.0
disallow: /
user-agent: Ahrefs-Bot/3.0
disallow: /
user-agent: Ahrefs-Bot/4.0
disallow: /
user-agent: Ahrefs-Bot/5.0
disallow: /
user-agent: aiHitBot
disallow: /
user-agent: aiHitBot/2.9
disallow: /
user-agent: Anonymous/0.0
disallow: /
user-agent: Arachnida
disallow: /
user-agent: Associative Spider
disallow: /
User-agent: Baiduspider
disallow: /
User-agent: Baidu Spider
disallow: /
User-agent: Battleztar Bazinga
disallow: /
User-agent: Battleztar Bazinga/0.01
disallow: /
User-agent: BDFetch
disallow: /
User-agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
Disallow: /
User-agent: ZmEu
Disallow: /
User-agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
Disallow: /
User-agent: betaBot
disallow: /
User-agent: bieshu
disallow: /
User-agent: Bigli SEO
disallow: /
User-agent: Blackboard Safeassign
disallow: /
User-agent: Blazer 1.0
disallow: /
User-agent: BLEXBot
disallow: /
User-agent: BLEXBot/1.0
disallow: /
User-agent: BLP_bbot
disallow: /
User-agent: BLP_bbot/0.1
disallow: /
User-agent: BOIA-Accessibility-Agent/PR 1.0
disallow: /
User-agent: BOT for JCE
disallow: /
User-agent: BOT/0.1 (BOT for JCE)
disallow: /
User-agent: BPImageWalker
disallow: /
User-agent: BPImageWalker/2.0
disallow: /
User-agent: BUbiNG
disallow: /
User-agent: BuiBui-Bot
disallow: /
User-agent: BuiBui-Bot/1.0
disallow: /
User-agent: ca-crawler
disallow: /
User-agent: ca-crawler/1.0
disallow: /
User-agent: CakePHP
disallow: /
User-agent: Calypso v/0.01
disallow: /
User-agent: Calypso
disallow: /
User-agent: CB/Nutch-1.7
disallow: /
User-agent: CCBot
disallow: /
User-agent: CCBot/2.0
disallow: /
User-agent: Checkbot
disallow: /
User-agent: checkgzipcompression.com
disallow: /
User-agent: chushou
disallow: /
User-agent: CloudServerMarketSpider
disallow: /
User-agent: CloudServerMarketSpider/1.0
disallow: /
User-agent: Clushbot/3.x-BinaryFury
disallow: /
User-agent: CMS Crawler
disallow: /
User-agent: CMS Crawler: http://www.cmscrawler.com
disallow: /
User-agent: coccoc
disallow: /
User-agent: CoinCornerBot
disallow: /
User-agent: CoinCornerBot/1.1
disallow: /
User-agent: Copyscape
disallow: /
User-agent: crawler4j
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.0
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.1
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.7
disallow: /
User-agent: CrazyWebCrawler
disallow: /
User-agent: CrazyWebCrawler-Spider
disallow: /
User-agent: Crowsnest
disallow: /
User-agent: Crowsnest/0.5
disallow: /
User-agent: Curious George - www.analyticsseo.com/crawler
disallow: /
User-agent: Curious George
disallow: /
User-agent: cuwhois
disallow: /
User-agent: cuwhois/1.0
disallow: /
User-agent: dahoms
disallow: /
User-agent: datagnionbot
disallow: /
User-agent: DeuSu/5.0.2
disallow: /
User-agent: Digincore
disallow: /
User-agent: Digincore bot
disallow: /
User-agent: Dispatch/0.11.0
disallow: /
User-agent: Domain Re-Animator Bot
disallow: /
User-agent: DomainAppender /1.0
disallow: /
User-agent: DomainAppender
disallow: /
User-agent: DomainCrawler/3.0
disallow: /
User-agent: DomainSigmaCrawler
disallow: /
User-agent: DomainSigmaCrawler/0.1
disallow: /
User-agent: Domnutch
disallow: /
User-agent: Domnutch-Bot
disallow: /
User-agent: Domnutch-Bot/Nutch
disallow: /
User-agent: Domnutch-Bot/Nutch-1.0
disallow: /
User-agent: dotbot
disallow: /
User-agent: ECCP/1.2.1
disallow: /
User-agent: eCommerceBot
disallow: /
User-agent: enlle punto com/Nutch-1.9
disallow: /
User-agent: EPiServer Link Checker
disallow: /
User-agent: EuripBot
disallow: /
User-agent: EuripBot/2.0
disallow: /
User-agent: evc/2.0
disallow: /
User-agent: evc-batch
disallow: /
User-agent: evc-batch/2.0
disallow: /
User-agent: Express WebPictures
disallow: /
User-agent: Faraday v0.8.8
disallow: /
User-agent: Faraday
disallow: /
User-agent: Findxbot
disallow: /
User-agent: Findxbot/1.0
disallow: /
User-agent: Flamingo_SearchEngine
disallow: /
User-agent: Flipboard Robot
disallow: /
User-agent: GetProxi.es-bot
disallow: /
User-agent: GetProxi.es-bot/1.1
disallow: /
User-agent: GigablastOpenSource
disallow: /
User-agent: GigablastOpenSource/1.0
disallow: /
User-agent: Girafabot
disallow: /
User-agent: Gluten Free Crawler
disallow: /
User-agent: Gluten Free Crawler/1.0
disallow: /
User-agent: GriffinBot
disallow: /
User-agent: GrifinBot/0.01
disallow: /
User-agent: GWPImages
disallow: /
User-agent: GWPImages/1.0
disallow: /
User-agent: Haiula
disallow: /
User-agent: Haiula/1.4
disallow: /
User-agent: HaosouSpider
disallow: /
User-agent: Hivemind
disallow: /
User-agent: HostHarvest
disallow: /
User-agent: HostHarvest/0.4.28
disallow: /
User-agent: HRCrawler
disallow: /
User-agent: HRCrawler/2.0
disallow: /
User-agent: http://git.io/tl_S2w
disallow: /
User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
disallow: /
User-agent: HubSpot Links Crawler 1.0
disallow: /
User-agent: HubSpot Webcrawler
disallow: /
User-agent: HubSpot
disallow: /
User-agent: hunchan
disallow: /
User-agent: HyperCrawl
disallow: /
User-agent: HyperCrawl/0.2
disallow: /
User-agent: ICAP-IOD
disallow: /
User-agent: ICC-Crawler
disallow: /
User-agent: ICC-Crawler/2.0
disallow: /
User-agent: Ichiro Robot
disallow: /
User-agent: image.coccoc/1.0
disallow: /
User-agent: Image2play
disallow: /
User-agent: Image2play/0.1
disallow: /
User-agent: Indy Library
disallow: /
User-agent: InsightsCollector
disallow: /
User-agent: InsightsCollector/0.1
disallow: /
User-agent: InsightsCollector/0.1beta
disallow: /
User-agent: integrity/5
disallow: /
User-agent: InterNaetBoten
disallow: /
User-agent: InterNaetBoten/0.99
disallow: /
User-agent: Barkrowler/0.7 (+http://www.exensa.com/crawl)
disallow: /
User-agent: Barkrowler
disallow: /
User-agent: IRL Crawler
disallow: /
User-agent: James BOT - WebCrawler
disallow: /
User-agent: James BOT
disallow: /
User-agent: JamesBOT
disallow: /
User-agent: JetBrains 5.0
disallow: /
User-agent: JetBrains
disallow: /
User-agent: Kraken
disallow: /
User-agent: Kraken/0.1
disallow: /
User-agent: Kyoto-Tohoku-Crawler/v1
disallow: /
User-agent: larbin
disallow: /
User-agent: lechenie
disallow: /
User-agent: libwww-perl
disallow: /
User-agent: link checker
disallow: /
User-agent: Link/1.0
disallow: /
User-agent: linkCheck
disallow: /
User-agent: linkCheckV3.0
disallow: /
User-agent: Linkdex
disallow: /
User-agent: linkdex.com/v2.0
disallow: /
User-agent: linkdex.com/v2.1
disallow: /
User-agent: LinkdexBot
disallow: /
User-agent: linkdexbot/2.0
disallow: /
User-agent: linkdexbot/2.1
disallow: /
User-agent: linkdexbot-mobile/2.1
disallow: /
User-agent: LinkpadBot
disallow: /
User-agent: LinkpadBot/1.06
disallow: /
User-agent: LinqiaScrapeBot
disallow: /
User-agent: LinqiaScrapeBot/1.0
disallow: /
User-agent: Lipperhey SEO Service
disallow: /
User-agent: Lipperhey
disallow: /
User-agent: Lipperhey-Kaus-Australis
disallow: /
User-agent: Lipperhey-Kaus-Australis/5.0
disallow: /
User-agent: listicka
disallow: /
User-agent: LSSRocketCrawler
disallow: /
User-agent: LSSRocketCrawler/1.0 LightspeedSystems
disallow: /
User-agent: LSSRocketCrawler/1.0
disallow: /
User-agent: ltx71
disallow: /
User-agent: LWNutch/Nutch-1.4
disallow: /
User-agent: Mail.RU
disallow: /
User-agent: Mail.RU_Bot
disallow: /
User-agent: Mail.RU_Bot/2.0
disallow: /
User-agent: Mail.RU_Bot/Fast/2.0
disallow: /
User-agent: md5sum
disallow: /
User-agent: md5sum\x22
disallow: /
User-agent: meanpathbot
disallow: /
User-agent: MegaIndex.ru
disallow: /
User-agent: MegaIndex.ru/2.0
disallow: /
User-agent: mezhpozvonochnoi
disallow: /
User-agent: Mike-Crawler
disallow: /
User-agent: MixBot
disallow: /
User-agent: MixrankBot
disallow: /
User-agent: MJ12bot
disallow: /
User-agent: Monkeybot/0.1
disallow: /
User-agent: my crawler
disallow: /
User-agent: My Nutch Spider/Nutch-1.9
disallow: /
User-agent: mycrowl/Nutch-1.9
disallow: /
User-agent: MyGreatUA/2.0
disallow: /
User-agent: MyIPTest
disallow: /
User-agent: NameProtect Robot
disallow: /
User-agent: NerdyBot
disallow: /
User-agent: Netcraft Spider
disallow: /
User-agent: netEstate NE Crawler
disallow: /
User-agent: NetLyzer FastProbe
disallow: /
User-agent: NetResearchServer
disallow: /
User-agent: NetResearchServer/4.0
disallow: /
User-agent: Nmap Scripting Engine
disallow: /
User-agent: node.io
disallow: /
User-agent: node.js
disallow: /
User-agent: Node/simplecrawler 0.5.2
disallow: /
User-agent: Node/simplecrawler
disallow: /
User-agent: oBot/2.3.1
disallow: /
User-agent: omgilibot
disallow: /
User-agent: omgilibot/0.4
disallow: /
User-agent: Online Domain Tools - Online Website Link Checker
disallow: /
User-agent: Online Domain Tools - Online Website Link Checker/1.2
disallow: /
User-agent: Openfind Robot
disallow: /
User-agent: OpenHoseBot
disallow: /
User-agent: OpenHoseBot/2.1
disallow: /
User-agent: Openstat
disallow: /
User-agent: Openstat/0.1
disallow: /
User-agent: OptimizationCrawler
disallow: /
User-agent: OptimizationCrawler/0.2
disallow: /
User-agent: Page Analyzer v4.0
disallow: /
User-agent: Page Analyzer
disallow: /
User-agent: PageAnalyzer
disallow: /
User-agent: PageAnalyzer/1.1
disallow: /
User-agent: PageAnalyzer/1.5
disallow: /
User-agent: PagesInventory
disallow: /
User-agent: Pagespeed/1.1 Fetcher
disallow: /
User-agent: Pagespeed/1.1
disallow: /
User-agent: Pagespeedbot
disallow: /
User-agent: Perl LWP
disallow: /
User-agent: PHPCrawl
disallow: /
User-agent: phpSiteCheck 1.0
disallow: /
User-agent: phpSiteCheck
disallow: /
User-agent: Plukkie
disallow: /
User-agent: POGS/2.0
disallow: /
User-agent: Powermarks
disallow: /
User-agent: PowerPivot
disallow: /
User-agent: PRIVACY_ENFAQ.jsp
disallow: /
User-agent: Prlog
disallow: /
User-agent: Prlog/1.0
disallow: /
User-agent: publiclibraryarchive.org
disallow: /
User-agent: publiclibraryarchive.org/1.0
disallow: /
User-agent: Pu_iN Crawler
disallow: /
User-agent: Putin
disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /
User-agent: Putin spider
disallow: /
User-agent: qingdao
disallow: /
User-agent: QlikView
disallow: /
User-agent: quipu
disallow: /
User-agent: quipu/1.0
disallow: /
User-agent: quipu/2.0
disallow: /
User-agent: R6_CommentReader
disallow: /
User-agent: R6_FeedFetcher
disallow: /
User-agent: Riddler
disallow: /
User-agent: RivalSeek.com-Bot
disallow: /
User-agent: rogerbot
disallow: /
User-agent: rogerbot/1.0
disallow: /
User-agent: rootlink
disallow: /
User-agent: RU_Bot/2.0
disallow: /
User-agent: Scopia
disallow: /
User-agent: Scopia crawler
disallow: /
User-agent: Scopia crawler 1.0
disallow: /
User-agent: Scopia crawler 1.1
disallow: /
User-agent: Scopia crawler 1.2
disallow: /
User-agent: Scrapy
disallow: /
User-agent: Scrapy/0.16.5
disallow: /
User-agent: Scrapy/0.24.4
disallow: /
User-agent: Scrapy/0.24.5
disallow: /
User-agent: Scrapy/0.24.6
disallow: /
User-agent: Scrapy/1.0.1
disallow: /
User-agent: Screaming Frog SEO Spider
disallow: /
User-agent: Screaming Frog SEO Spider/2,55
disallow: /
User-agent: Screaming Frog SEO Spider/2.55
disallow: /
User-agent: Screaming Frog SEO Spider/3.1
disallow: /
User-agent: Screaming Frog SEO Spider/3.3
disallow: /
User-agent: Screaming Frog SEO Spider/4.1
disallow: /
User-agent: Screaming Frog SEO Spider/5.0
disallow: /
User-agent: Screaming Frog SEO Spider/5.1
disallow: /
User-agent: Screaming Frog SEO Spider/5.1 Beta 2
disallow: /
User-agent: scrutiny/4
disallow: /
User-agent: SemrushBot
disallow: /
User-agent: SemrushBot-SA
disallow: /
User-agent: SEOdiver/1.0
disallow: /
User-agent: SEOkicks
disallow: /
User-agent: SEOkicks-Robot
disallow: /
User-agent: SEOlyticsCrawler
disallow: /
User-agent: SEOlyticsCrawler/3.0
disallow: /
User-agent: seoscanners
disallow: /
User-agent: seoscanners.net/1
disallow: /
User-agent: SEOstats 2.1.0
disallow: /
User-agent: Seosys/Nutch-2.3
disallow: /
User-agent: SetCronJob/1.0
disallow: /
User-agent: R6_FeedFetcher_(www.radian6.com/crawler)
disallow: /
User-agent: SeznamBot
disallow: /
User-agent: SheerBoredom.Experimental.Robot
disallow: /
User-agent: SheerBoredom.Experimental.Robot/0.2
disallow: /
User-agent: ShowyouBot
disallow: /
User-agent: Simplecrawler
disallow: /
User-agent: SISTRIX Crawler
disallow: /
User-agent: sistrix
disallow: /
User-agent: SiteBot
disallow: /
User-agent: SiteBot/0.1
disallow: /
User-agent: SiteExplorer
disallow: /
User-agent: SiteExplorer/1.0
disallow: /
User-agent: SiteExplorer/1.0b
disallow: /
User-agent: Siteluxbot
disallow: /
User-agent: Siteluxbot/1.0
disallow: /
User-agent: SkimBot
disallow: /
User-agent: RSSingBot (http://www.rssing.com)
disallow: /
User-agent: SkimBot/1.0
disallow: /
User-agent: sky nutch crawler/Nutch-1.9
disallow: /
User-agent: SMTBot
disallow: /
User-agent: SMTBot/1.0
disallow: /
User-agent: SNK Screenshot Bot
disallow: /
User-agent: SNK Screenshot Bot/0.20
disallow: /
User-agent: Sogou Spider
disallow: /
User-agent: Sogou web spider
disallow: /
User-agent: SpamBayes
disallow: /
User-agent: SpamBayes/1.1a3+
disallow: /
User-agent: spbot
disallow: /
User-agent: spbot/4.4.2
disallow: /
User-agent: spiderbot
disallow: /
User-agent: SpiderLing
disallow: /
User-agent: Spiderbot/Nutch-1.7
disallow: /
User-agent: spray-can
disallow: /
User-agent: spray-can/1.2.1
disallow: /
User-agent: SSG/3.0
disallow: /
User-agent: Statastico
disallow: /
User-agent: Statastico/4.0
disallow: /
User-agent: Steeler
disallow: /
User-agent: Steeler/3.5
disallow: /
User-agent: Stratagems Kumo
disallow: /
User-agent: Stratagems
disallow: /
User-agent: StudioFACA Search
disallow: /
User-agent: StudioFACA
disallow: /
User-agent: sukibot
disallow: /
User-agent: sukibot_heritrix
disallow: /
User-agent: sukibot_heritrix/3.1.1
disallow: /
User-agent: SurveyBot
disallow: /
User-agent: Synapse
disallow: /
User-agent: Synthesio Crawler release MonaLisa
disallow: /
User-Agent: tbot-nutch/Nutch-1.10
disallow: /
User-Agent: Traackr.com bot
disallow: /
User-Agent: trendictionbot
disallow: /
User-Agent: Trendiction-Bot
disallow: /
User-Agent: TrueBot
disallow: /
User-Agent: TrueBot/1.0
disallow: /
User-Agent: TulipChain/5.xx
disallow: /
User-Agent: TWMBot/0.1
disallow: /
User-Agent: Typhoeus
disallow: /
User-Agent: UCMore Crawler App
disallow: /
User-Agent: uMBot-LN
disallow: /
User-Agent: uMBot-LN/1.0
disallow: /
User-Agent: updown_tester
disallow: /
User-Agent: URLChecker
disallow: /
User-agent: V1.0/1.2
disallow: /
User-agent: w3af.org
disallow: /
User-agent: WASALive
disallow: /
User-agent: WASALive-Bot
disallow: /
User-agent: vBSEO
disallow: /
User-agent: WBSearchBot
disallow: /
User-agent: WBSearchBot/1.1
disallow: /
User-agent: WeAreNotEvil
disallow: /
User-agent: WebAlta
disallow: /
User-agent: WebAlta Crawler
disallow: /
User-agent: Web corpus crawler
disallow: /
User-agent: WebCookies
disallow: /
User-agent: WebCookies/1.0
disallow: /
User-agent: WebCopier vx.xa
disallow: /
User-agent: Webnest 0.9
disallow: /
User-agent: WebQL
disallow: /
User-agent: Webscout
disallow: /
User-agent: Webscout/1.0
disallow: /
User-agent: Web-sniffer
disallow: /
User-agent: Web-sniffer/1.1.0
disallow: /
User-agent: Webster Pro V3.4
disallow: /
User-agent: WebTarantula.com Crawler
disallow: /
User-agent: WeCrawlForThePeace
disallow: /
User-agent: VegeBot
disallow: /
User-agent: Vegi bot
disallow: /
User-agent: WeLikeLinks
disallow: /
User-agent: VeriCiteCrawler
disallow: /
User-agent: VeriCiteCrawler/Nutch-1.9
disallow: /
User-agent: WhatWeb
disallow: /
User-agent: WhatWeb/0.4.8-dev
disallow: /
User-agent: Visited by http://tools.geek-tools.org
disallow: /
User-agent: voltron
disallow: /
User-agent: woobot
disallow: /
User-agent: woobot/1.1
disallow: /
User-agent: woobot/2.0
disallow: /
User-agent: Vorboss Web Crawler
disallow: /
User-agent: Vorboss Web Crawler/Nutch-2.3
disallow: /
User-agent: WorldBrewBot
disallow: /
User-agent: WorldBrewBot/2.1
disallow: /
User-agent: worldwebheritage.org
disallow: /
User-agent: worldwebheritage.org/1.0
disallow: /
User-agent: wscheck.com
disallow: /
User-agent: wscheck.com/1.0.0
disallow: /
User-agent: www.deadlinkchecker.com
disallow: /
User-agent: www.petitsage.fr site detector 0.4
disallow: /
User-agent: WWW-Mechanize
disallow: /
User-agent: WWW-Mechanize/1.74
disallow: /
User-agent: Xenu Link Sleuth
disallow: /
User-agent: Xenu's Link Sleuth
disallow: /
User-agent: XoviBot
disallow: /
User-agent: XoviBot/2.0
disallow: /
User-agent: XSpider
disallow: /
User-agent: Yandex Robot
disallow: /
User-agent: Yandex
disallow: /
User-agent: Yetibot
disallow: /
User-agent: YisouSpider
disallow: /
User-agent: yoozBot
disallow: /
User-agent: yoozBot-2.2
disallow: /
User-agent: zgrab/0.x
disallow: /
User-agent: zzabmbot
disallow: /
User-agent: zzabmbot/1.0
disallow: /
User-agent: Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-2.3)
Disallow: /
User-agent: Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Disallow: /
User-agent: Gigabot/2.0
Disallow: /
User-agent: GameSpyHTTP/1.0
Disallow: /
User-agent: Baiduspider ( http://www.baidu.com/search/spider.htm)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Disallow: /
User-Agent: istellabot/t.1.13
Disallow: /
User-Agent: MagpieRSS/0.72 (+http://magpierss.sf.net)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /
User-Agent: Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
Disallow: /
User-Agent: Liferea
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-Agent: ahrefsbot
Disallow: /
User-Agent: Ahrefs
Disallow: /
User-Agent: Ahrefsbot
Disallow: /
User-Agent: BoardReader Blog Indexer(http://boardreader.com)
Disallow: /
User-Agent: BoardReader
Disallow: /
User-Agent: YOUBot
Disallow: /
User-Agent: AhrefsBot.com
Disallow: /
User-Agent: yandexBot
Disallow: /
User-Agent: Dow Jones Searchbot
Disallow: /
User-Agent: TurnitinBot
Disallow: /
User-Agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /
User-Agent: Almaden
Disallow: /
User-Agent: eCairn-Grabber/1.0
Disallow: /
User-Agent: eCairn-Grabber
Disallow: /
User-Agent: Amazon
Disallow: /
User-Agent: Gluten Free Crawler/1.0
Disallow: /
User-Agent: DotBot/1.1
Disallow: /
User-Agent: LinqiaCrawlerBot/1.0
Disallow: /
User-Agent: istellabot/t.1.13
Disallow: /
User-Agent: nutch-1.4/Nutch-1.4
Disallow: /
User-Agent: nutch-1.4
Disallow: /
User-Agent: DoBot
Disallow: /
User-Agent: AsapBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: coccocbot-web
Disallow: /
User-Agent: amazonbot
Disallow: /
User-Agent: Amazon-bot
Disallow:/
User-Agent: Meditoolkitbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Disallow: /
User-agent: Yeti/1.1
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: Yahoo! Slurp
Disallow: /
User-agent: HaoSouSpider
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Applebot
Disallow: /
User-agent: Applebot/0.1; +http://www.apple.com/go/applebot
Disallow: /
User-agent: Applebot/0.1
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Naverbot
Disallow: /
User-agent: seznambot
Disallow: /
User-agent: teoma
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: EtaoSpider
Disallow: /
User-Agent: swebot
Disallow: /
User-Agent: Newsbrain
Disallow: /
User-Agent: Feedreader 3.14
Disallow: /
User-Agent: Feedreader
Disallow: /
User-Agent: Feedreader 3.14 (Powered by Newsbrain)
Disallow: /
User-agent: aihitbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Exabot-Thumbnails
Disallow: /
User-agent: sitebot
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: TwengaBot-Discover
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Bender
Disallow: /
User-agent: discobot
Disallow: /
User-Agent: panscient.com
Disallow: /
User-Agent: searchwebengine.net
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: NerdByNature.Bot
disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: sistrix
Disallow: /
User-agent: SindiceBot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: IWAgent/1.0
Disallow: /
User-agent: IWAgent
Disallow: /
User-agent: BotALot
Disallow:/
User-agent: findfiles.net
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Goodzer
Disallow: /
User-agent: DJEcoBot
Disallow: /
User-agent: DJEcoBot/1.0
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
User-agent: AhrefsBot/5.2
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: spray-can/1.3.4
Disallow: /
User-agent: spray-can
Disallow: /
User-agent: spray can
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: 360spider
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: BacklinkCrawler
Disallow: /
User-agent: lemurwebcrawler
Disallow: /
User-agent: gigabot
Disallow: /
User-agent: discobot Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: FAST Enterprise Crawler 6
Disallow: /
User-agent: Sensis.com.au Web Crawler
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: XilopixBot/0.1
Disallow: /
User-agent: XilopixBot
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Arachmo
Disallow: /
User-agent: Acoon
Disallow: /
User-agent: adbeat_bot
Disallow: /
User-agent: AgentLinkSpammer
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: worio bot heritrix
Disallow: /
User-agent: trovitBot
Disallow: /
User-agent: MojeekBot/0.6
Disallow: /
User-agent: rogerbot/1.0
Disallow: /
User-agent: istellabot/t.1.13
Disallow: /
User-agent: SeznamBot/3.2
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: careerbot
Disallow: /
User-agent: SogouBot
Disallow: /
User-agent: Sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogouspider
Disallow: /
User-agent: Qihoo360Bot
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: COMODOSpider
Disallow: /
User-agent: findlinks
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: KaloogaBot
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: Mail.RU
Disallow: /
User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Disallow: /
User-agent: SurdotlyBot/1.0
Disallow: /
User-agent: SurdotlyBot
Disallow: /
User-agent: Apexoo Spider (http://www.apexoo.com/spider/)
Disallow: /
User-agent: Apexoo Spider 1.0
Disallow: /
User-agent: Apexoo Spider 1.1
Disallow: /
User-agent: AportCatalogRobot/2.0
Disallow: /
User-agent: Apostrophe/0.2.1.93 (compatible) filecache
Disallow: /
User-agent: Apostrophe
Disallow: /
User-agent: Aplix HTTP/1.0.1
Disallow: /
User-agent: moget
Disallow: /
User-agent: Ichiro
Disallow: /
User-agent: Ichiro 3.0
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: NetcraftSurveyAgent/1.0
Disallow: /
User-agent: OpenWebIndex/Nutch-1.6
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: panoptaStudyBot
Disallow: /
User-agent: checks.panopta.com
Disallow: /
User-agent: psbot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: SeznamBot/1.0
Disallow: /
User-agent: SeznamBot/1.1
Disallow: /
#SeznamBot/3.0
User-agent: SeznamBot/3.0
Disallow: /
User-agent: SistrixCrawler
Disallow: /
User-agent: Sistrix
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SISTRIX
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sosospider+
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sosospider/2.0
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: Wada.vn
Disallow: /
User-agent: Wada.vn Vietnamese Search
Disallow: /
User-agent: Wada.vn Vietnamese Search/2.1
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: YandexBot/3.0-MirrorDetector
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: YandexSomething
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: YoudaoBot/1.0
Disallow: /
User-agent: Zao
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs
Disallow: /
User-agent: aipbot
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: asterias
Disallow:/
User-agent: Aqua_Products
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: becomebot
Disallow: /
User-agent: Black Hole
Disallow:/
User-agent: BlowFish
Disallow:/
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite
Disallow:/
User-agent: CherryPickerSE
Disallow:/
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: FairAd Client
Disallow: /
User-agent: Fasterfox
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow:/
User-agent: Gaisbot
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: Harvest
Disallow:/
User-agent: Go!Zilla
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: HTTrack
Disallow: /
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: Iron33
Disallow: /
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density
Disallow:/
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: Mata Hari
Disallow: /
User-agent: MIIxpc
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NetMechanic
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: Oracle Ultra Search
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: ProPowerBot
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: psbot
Disallow: /
User-agent: QueryN Metasearch
Disallow:/
User-agent: Radiation Retriever
Disallow: /
User-agent: RepoMonkey
Disallow:/
User-agent: Bait
Disallow:/
User-agent: RMA
Disallow:/
User-agent: searchpreview
Disallow: /
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz
Disallow:/
User-agent: SurveyBot
Disallow: /
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: turingos
Disallow:/
User-agent: TurnitinBot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URLy Warning
Disallow:/
User-agent: URL_Spider_Pro
Disallow: /
User-agent: VCI
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebCapture
Disallow: /
User-agent: scalaj-http/1.0
Disallow: /
User-agent: WebCollage
Disallow: /
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: Wget
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
User-agent: Meltawer
Disallow: /
User-agent: Digimind
Disallow: /
User-agent: Knowings
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: Cision
Disallow: /
User-agent: Talkwater
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: kbcrawl
Disallow: /
User-agent: AmiSoftware
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: Ask n read
Disallow: /
User-agent: Qwam content intelligence
Disallow: /
User-agent: Zite
Disallow: /
User-agent: flipboard
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Youmag
Disallow: /
User-agent: Synthesio
Disallow: /
User-agent: trendybuzz
Disallow: /
User-agent: spotter
Disallow: /
User-agent: scoop.it
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: 5emeRue
Disallow: /
User-agent: Augure
Disallow: /
User-agent: Corporama
Disallow: /
User-agent: readability.com
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver-web.archive.org
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: libwww
Disallow: /
User-agent: wget
Disallow: /
User-agent: 5erue
Disallow: /
User-agent: Sysomos/1.0
Disallow: /
User-agent: adequat
Disallow: /
User-agent: adequat-systems
Disallow: /
User-agent: auramundi
Disallow: /
User-agent: coexel
Disallow: /
User-agent: ellisphere
Disallow: /
User-agent: leadbox
Disallow: /
User-agent: mention
Disallow: /
User-agent: Moreover
Disallow: /
User-agent: mytwip
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: mixnode.com
Disallow: /
User-agent: opinion-tracker
Disallow: /
User-agent: proxem
Disallow: /
User-agent: score3
Disallow: /
User-agent: trendeo
Disallow: /
User-agent: vecteurplus
Disallow: /
User-agent: verticalsearch
Disallow: /
User-agent: vsw
Disallow: /
User-agent: winello
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: infoseek
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: asknread.com
Disallow: /
User-agent: ellisphere
Disallow: /
User-agent: spotter
Disallow: /
User-agent: LexxeBot/1.0
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; spbot/2.0; http://www.seoprofiler.com/bot/ )
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Nuzzel
Disallow: /
User-agent: Nuzzel Bot
Disallow: /
User-agent: Sosospider+(+http://help.soso.com/webspider.htm)
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: SiteBot/0.1
Disallow: /
User-agent: SiteBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: crystalsemanticsbot
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: NetSeer crawler
Disallow: /
User-agent: trovitBot
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: discobot
Disallow: /
User-agent: Jyxobot
Disallow: /
User-agent: sogou
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogoubot
Disallow: /
User-agent: Sogou.com
Disallow: /
User-agent: sogouspider-106-120-173-138.crawl.sogou.com
Disallow: /
User-agent: crawl13.exabot.com
Disallow: /
User-agent: sogouspider-106-120-173-138
Disallow: /
User-agent: Sogou web spider/4.0
Disallow: /
User-agent: Sogouwebspider
Disallow: /
User-agent: sogouspider
Disallow: /
User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Disallow: /
User-agent: Exabot/3.0
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: sistrix
Disallow: /
User-agent: heritrix
Disallow: /
User-agent: GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: NerdByNature.Bot
Disallow: /
User-agent: psbot
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: AddThis.com robot tech.support@clearspring.com
Disallow: /
User-agent: AddThis.com
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: proximic
Disallow: /
User-agent: discoverybot
Disallow: /
User-agent: bl.uk_lddc_bot
Disallow: /
User-agent: IstellaBot
Disallow: /
User-agent: seokicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: Bender
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Yasni
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: netEstate NE Crawler
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Pixray-Seeker
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: integromedb
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: BDCbot
Disallow: /
User-agent: GrapeshotCrawler
Disallow: /
User-agent: WeSEE:Search
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: admantx
Disallow: /
User-agent: spbot
Disallow: /
User-agent: BUbiNG
Disallow: /
User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: Harvest/1.5
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Nutch
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: Spiderbot
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: Xenu Link Sleuth/1.3.8
User-agent: yacy
User-agent: yandex
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Spiderbot/Nutch-1.7
User-agent: g2reader-bot/1.0
User-agent: SocialShare/1.0
User-agent: Owlin Bot
User-agent: trendictionbot
Disallow: /
# Disallow: Sistrix
User-agent: sistrix
Disallow: /
# Disallow: Sistrix
User-agent: SISTRIX Crawler
Disallow: /
# Disallow: Sistrix
User-agent: SISTRIX
Disallow: /
# Disallow: SEOkicks-Robot
User-agent: SEOkicks-Robot
Disallow: /
# Disallow: jobs.de-Robot
User-agent: jobs.de-Robot
Disallow: /
# Backlink Analysis
user-agent: AhrefsBot
disallow: /
# Bot der Leipziger Unister Holding GmbH
user-agent: UnisterBot
disallow: /
# http://moz.com/products
User-agent: dotbot
Disallow: /
# http://www.searchmetrics.com
User-agent: SearchmetricsBot
Disallow: /
# http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
User-agent: MJ12bot
Disallow: /
# http://www.domaintools.com/webmasters/surveybot.php
User-agent: SurveyBot
Disallow: /
# http://www.seodiver.com/bot
user-agent: SEOdiver
disallow: /
# http://openlinkprofiler.org/bot
User-agent: spbot
Disallow: /
# http://www.wotbox.com/bot/
User-agent: wotbox
Disallow: /
# http://www.opensiteexplorer.org/dotbot
# http://moz.com/researchtools/ose/dotbot
User-agent: dotbot
Disallow: /
# http://www.meanpath.com/meanpathbot.html
User-agent: meanpathbot
Disallow: /
# http://www.backlinktest.com/crawler.html
User-agent: BacklinkCrawler
Disallow: /
# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler
Disallow: /
# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Disallow: /
# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1
Disallow: /
# http://filterdb.iss.net/crawler/
User-agent: oBot
Disallow: /
User-agent: fr-crawler
Disallow: /
# http://webmeup-crawler.com
User-agent: BLEXBot
Disallow: /
# https://megaindex.com/crawler
User-agent: MegaIndex.ru
Disallow: /
User-agent: megaindex.com
Disallow: /
# http://www.cloudservermarket.com
User-Agent: CloudServerMarketSpider
Disallow: /
# http://www.trendiction.de/de/publisher/bot
User-Agent: trendictionbot
Disallow: /
# http://www.exalead.com
User-agent: Exabot
Disallow: /
# http://www.career-x.de/bot.html
User-agent: careerbot
Disallow: /
# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /
User-agent: seoscanners.net/1
Disallow: /
User-agent: MetaJobBot
Disallow: /
User-agent: Spiderbot
Disallow: /
User-agent: LinkStats
Disallow: /
User-agent: JobboerseBot
Disallow: /
User-agent: ICCrawler
Disallow: /
User-agent: Plista
Disallow: /
User-agent: Domain Re-Animator Bot
Disallow: /
User-agent: Aboundexbot
Disallow: /
User-agent: acapbot
Disallow: /
User-agent: AdnormCrawler
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: AOLbot/4.0
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: Attribot
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: Bot.AraTurka.com
Disallow: /
User-agent: Butterfly/1.0
Disallow: /
User-agent: BuzzSumo
Disallow: /
User-agent: cis455crawler
Disallow: /
User-agent: ClarityBot
Disallow: /
User-agent: ClarityDailyBot
Disallow: /
User-agent: CMS Crawler
Disallow: /
User-agent: CommonCrawler
Disallow: /
User-agent: crawler4j (http://code.google.com/p/crawler4j/)
Disallow: /
User-agent: CrazyWebCrawler
Disallow: /
User-agent: CrazyWebCrawler-Spider
Disallow: /
User-agent: DataparkSearch
Disallow: /
User-agent: deepcrawl
Disallow: /
User-agent: demandbase-bot
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: dialogsearch.com
Disallow: /
User-agent: diffbot
Disallow: /
User-agent: dj-research
Disallow: /
User-agent: DomainAppender
Disallow: /
User-agent: DomainMacroCrawler
Disallow: /
User-agent: DomainSigmaCrawler
Disallow: /
User-agent: Domain Re-Animator Bot
Disallow: /
User-agent: Ezooms Robot
Disallow: /
User-agent: Experibot
Disallow: /
User-agent: FavOrg
Disallow: /
# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /
# https://turnitin.com/robot/crawlerinfo.html
User-agent: turnitinbot
Disallow: /
# http://help.coccoc.com/
User-agent: coccoc
Disallow: /
# ubermetrics-technologies.com
User-agent: um-IC
Disallow: /
User-agent: scanbot
Disallow: /
# datenbutler.de
User-agent: mindUpBot
Disallow: /
# http://searchgears.de/uber-uns/crawling-faq.html
User-agent: sg-Orbiter
Disallow: /
# http://commoncrawl.org/faq/
User-agent: CCBot
Disallow: /
# http://linkfluence.net/
User-agent: Kraken
Disallow: /
# http://www.botje.com/plukkie.htm
User-agent: plukkie
Disallow: /
# https://www.safedns.com/searchbot
User-agent: SafeDNSBot
Disallow: /
# http://www.haosou.com/help/help_3_2.html
User-agent: 360Spider
Disallow: /
# http://www.haosou.com/help/help_3_2.html
User-agent: HaosouSpider
Disallow: /
# http://www.moz.com/dp/rogerbot
User-agent: rogerbot
Disallow: /
# http://www.openhose.org/bot.html
User-agent: OpenHoseBot
Disallow: /
# http://www.screamingfrog.co.uk/seo-spider/
User-agent: Screaming Frog SEO Spider
Disallow: /
# http://thumbsniper.com
User-agent: ThumbSniper
Disallow: /
# http://www.radian6.com/crawler
User-agent: R6_CommentReader
Disallow: /
User-agent: ImplisenseBot
Disallow: /
# http://cliqz.com/company/cliqzbot
User-agent: Cliqzbot
Disallow: /
# https://www.aihitdata.com/about
User-agent: aiHitBot
Disallow: /
# http://www.trendiction.com/en/publisher/bot
User-Agent: trendictionbot
Disallow: /
# Riddler
# Riddler (http://riddler.io/about)
# http://riddler.io/about
User-agent: Riddler
Disallow: /
# SMTBot
# http://www.similartech.com/smtbot
# Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
User-agent: SMTBot
Disallow: /
# ScoutJet
# http://www.scoutjet.com
# Mozilla/5.0+(compatible;+ScoutJet;++http://www.scoutjet.com/)
# Mozilla/5.0+(compatible;+Blekkobot;+ScoutJet;++http://blekko.com/about/blekkobot)
User-agent: ScoutJet
Disallow: /
# SeeBot
# http://www.seegnify.com/bot
# seebot/2.0 (+http://www.seegnify.com/bot)
User-agent: SeeBot
Disallow: /
# Archive.org Bot
# http://archive.org/details/archive.org_bot
# Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot)
# Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot)
User-agent: archive.org_bot
Disallow: /
# AdBeat Bot
# https://www.adbeat.com/operation_policy
# adbeat_bot
User-agent: adbeat_bot
Disallow: /
# MeanPathBot
# http://www.meanpath.com/meanpathbot.html
# Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
User-agent: meanpathbot
Disallow: /
# Mojeek Bot
# https://www.mojeek.com/bot.html
# Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /
# Speedy Spider / EntireWeb Search
# http://www.entireweb.com
# Speedy+Spider+(http://www.entireweb.com)
User-agent: Speedy Spider
Disallow: /
# GenieO
# http://www.genieo.com/webfilter.html
# Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
User-agent: genieo
Disallow: /
# scrapy.org
# by default ROBOTSTXT_OBEY is set to false
# http://doc.scrapy.org/en/latest/topics/settings.html
User-agent: Scrapy
Disallow: /
# Gigablast Open Source Search Engine/Crawler/Spider
# You can ignore robots.txt
# http://gigablast.com/admin.html
# GigablastOpenSource/1.0
User-agent: GigablastOpenSource
Disallow: /
# Semrush Bot
# http://www.semrush.com/bot.html
# Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html)
User-agent: SemrushBot
Disallow: /
# Proximic
# http://www.proximic.com/info/spider.php
# Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
User-agent: proximic
Disallow: /
# AppleBot
# Used by Siri/Spotlight suggestions (commented out for now)
# http://www.apple.com/go/applebot
# Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot)
# Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot)
# Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot)
User-agent: applebot
Disallow: /
# CCBot
# http://commoncrawl.org/faq/
# CCBot/2.0+(http://commoncrawl.org/faq/)
User-agent: CCBot
Disallow: /
# Plukkie
# http://www.botje.com/plukkie.htm
# Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
User-agent: Plukkie
Disallow: /
# Lipperhey Kaus Australis
# https://www.lipperhey.com/en/about/
# Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
User-agent: lipperhey
Disallow: /
# oBot
# http://filterdb.iss.net/crawler/
# Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
# Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
User-agent: oBot
Disallow: /
# Screaming Frog SEO Spider
# Downloaded software, robots.txt can be ignored
# http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
# Screaming+Frog+SEO+Spider/2.55
# Screaming+Frog+SEO+Spider/3.3
User-agent: Screaming Frog SEO Spider
Disallow: /
# VegeBot
# http://www.exactbot.com/vegebot/index.html
User-agent: VegeBot
Disallow: /
# ScreenerBot
# ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
User-agent: ScreenerBot
Disallow: /
# Garlik from Experian
# GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
User-agent: Garlik
Disallow: /
# BusinessBot: Nathan@lead-caddy.com
User-agent: BusinessBot
Disallow: /
# FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
User-agent: FAST Enterprise
Disallow: /
# Nutch
# http://nutch.apache.org/bot.html
# Nutch+Spider/Nutch-1.5
# mycrowl/Nutch-1.9
# nutch+crawler/Nutch-1.9
# sky+nutch+crawler/Nutch-1.9
# Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
User-agent: Nutch
Disallow: /
# Open Link Profiler
# Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
# http://OpenLinkProfiler.org/bot
User-agent: spbot
Disallow: /
# James Bot
# Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
# http://cognitiveseo.com/bot.html
User-agent: JamesBOT
Disallow: /
# Kraken
# Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;+bot@linkfluence.net)
# http://linkfluence.net/
User-agent: Kraken
Disallow: /
# Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
# http://www.linkdex.com/bots/
User-agent: linkdex
Disallow: /
# Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
# Siteimprove.com
User-agent: SiteCheck-sitecrawl by Siteimprove.com
Disallow: /
# Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
# http://megaindex.com/crawler
User-agent: MegaIndex
Disallow: /
# Prlog
# Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
# http://prlog.ru/
User-agent: Prlog
Disallow: /
# Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
# https://www.aihitdata.com/about
User-agent: AiHitBot
Disallow: /
# KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
# http://www.kingland.com
User-agent: KinglandSystemsCorp
Disallow: /
# Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;+SitemapBot@stmp.com)
# http://www.sitemapbot.com
User-agent: SitemapBot
Disallow: /
# Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
# https://www.deepcrawl.com/bot
User-agent: deepcrawl
Disallow: /
# Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21
# http://www.alexa.com/toolbar
#User-agent: AlexaToolbar
#Disallow: /
# Mozilla/5.0+(compatible;+heritrix/3.3.0-SNAPSHOT-20150302-2206++http://127.0.0.1)
# https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
User-agent: heritrix
Disallow: /
# Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
# http://help.naver.com/robots/
User-agent: Yeti
Disallow: /
# Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,+contact+abuse@inpwrd.com+for+information)
User-agent: NetShelter
Disallow: /
# VSE/1.0+(rabraham@multiview.com)
# LSSRocketCrawler/1.0 LightspeedSystems
# http://lightspeedsystems.com
User-agent: LSSRocketCrawler
Disallow: /
# Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
# http://www.idmarch.org/bot.html
User-agent: idmarch
Disallow: /
# Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0
# Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0+(Following+Mediapartners-Google)
User-agent: Daumoa
Disallow: /
# Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
# https://www.companiesintheuk.co.uk/bot.html
User-agent: CukBot
Disallow: /
# Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
# http://www.expertsearch.nl/spider
User-agent: ExpertSearchSpider
Disallow: /
# Mozilla/5.0+(compatible;+Goodzer/2.0;+crawler@goodzer.com)
User-agent: Goodzer
Disallow: /
# WeSEE
# http://www.wesee.com/
User-agent: WeSEE
Disallow: /
# http://ltx71.com/
# ltx71 - (http://ltx71.com/)
User-agent: ltx71
Disallow: /
# rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,+rogerbot-crawler+shiny@moz.com)
# http://moz.com/help/pro/what-is-rogerbot-
User-agent: rogerbot
Disallow: /
# bitlybot
# http://bit.ly ??
User-agent: bitlybot
Disallow: /
# Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
# http://www.trendiction.de/bot
User-agent: trendictionbot
Disallow: /
# SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
User-agent: SearchMassive
Disallow: /
# spiderbot
User-agent: spiderbot
Disallow: /
# yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
# yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
User-agent: yacybot
Disallow: /
# AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;+help@aboutus.org)
# http://www.aboutus.org/Aboutus:Bot
User-agent: AboutUsBot
Disallow: /
# Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
# http://www.opensiteexplorer.org/dotbot
User-agent: DotBot
Disallow: /
# Mozilla/5.0+(compatible;+BLEXBot/1.0;++http://webmeup-crawler.com/)
# http://webmeup-crawler.com/
User-agent: BLEXBot
Disallow: /
# Xenu Link Sleuth 1.3.4
# Xenu Link Sleuth/1.3.8
User-agent: Xenu Link Sleuth
Disallow: /
# Go language
# Go+1.1+package+http
User-agent: Go 1.1
Disallow: /
# wotbox.com
# http://www.wotbox.com/bot/
# Wotbox/2.01+(+http://www.wotbox.com/bot/)
User-agent: Wotbox
Disallow: /
# yandex
# http://yandex.com/bots
# Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
User-agent: YandexBot
Disallow: /
# Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
# http://help.yahoo.com/help/us/ysearch/slurp
# User-agent: Yahoo! Slurp
# Disallow: /
# Yahoo Japan
# http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
# Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
User-agent: Y!J-ASR
Disallow: /
# domainreanimator.com
User-agent: Domain Re-Animator Bot
Disallow: /
# crazywebcrawler.com
User-agent: CrazyWebCrawler-Spider
Disallow: /
# crazywebcrawler.com
User-agent: CrazyWebCrawler
Disallow: /
# Dow Jones Searchbot
# Mozilla/5.0+(compatible;+Dow+Jones+Searchbot)
User-agent: Dow Jones SearchBot
Disallow: /
# OpenVAS.org
#User-agent: OpenVAS
#Disallow: /
# WeCrawlForThePeace - We are not Evil
User-agent: WeCrawForThePeace
Disallow: /
# WeLikeLinks - WeAreNotEvil
User-agent: WeLikeLinks
Disallow: /
# icevikatam/1.0
#User-agent: icevikatam
#Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Disallow: /
User-agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Disallow: /
User-agent: SocialRankIOBot; http://socialrank.io/about
Disallow: /
User-agent: SocialRankIOBot
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: Curious George
Disallow: /
User-agent: CybEye.com
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: ExB Language Crawler
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Genio
Disallow: /
User-agent: LWNutch
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: RediffNewsBot
Disallow: /
User-agent: SEOENGWorldBot
Disallow: /
User-agent: Scanmine
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShowyouBot
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: WocBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: daumoa
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: libcrawl
Disallow: /
User-agent: linkdex
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: repparser
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: sindice-site-manager
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: sogou
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: yolinkBot
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: wget
Disallow: /
# Block Sogou
User-agent: sogou spider
Disallow: /
# Block SEOkicks
User-agent: SEOkicks-Robot
Disallow: /
# Block BlexBot
User-agent: BLEXBot
Disallow: /
# Block SISTRIX
User-agent: SISTRIX Crawler
Disallow: /
# Block Uptime robot
User-agent: UptimeRobot/2.0
Disallow: /
User-agent: 008
Disallow: /
# Block Ezooms Robot
User-agent: Ezooms Robot
Disallow: /
# Block Perl LWP
User-agent: Perl LWP
Disallow: /
# Block BlexBot
User-agent: BLEXBot
Disallow: /
# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /
# Block WiseGuys Robot
User-agent: WiseGuys Robot
Disallow: /
# Block Turnitin Robot
User-agent: Turnitin Robot
Disallow: /
User-agent: HuaweiSymantecSpider
Disallow: /
User-agent: Purebot
Disallow: /
# Block Heritrix
User-agent: Heritrix
Disallow: /
User-agent: eContext/1.0 (eContext Classification Engine)
Disallow: /
User-agent: eContext/1.0
Disallow: /
User-agent: eContext
Disallow: /
# Block pricepi
User-agent: pimonster
Disallow: /
User-agent: Pimonster
Disallow: /
# Block Searchmetrics Bot
User-agent: SearchmetricsBot
Disallow: /
# Block Eniro
User-agent: ECCP/1.0 (search@eniro.com)
Disallow: /
# Block YandexBot
User-agent: Yandex
Disallow: /
# Block Baidu
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /
# Block SoGou
User-agent: Sogou Spider
Disallow: /
# Block Youdao
User-agent: YoudaoBot
Disallow: /
# Block MegaIndex.ru
User-agent: MegaIndex.ru/2.0
Disallow: /
User-agent: Yandex
Disallow: /
user-agent: AhrefsBot
disallow: /
User-Agent: trendictionbot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: search:robot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: xovi
Disallow: /
User-agent: Cutbot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: LivelapBot
Disallow: /
User-agent: LS Session
Disallow: /
User-agent: MagicBot
Disallow: /
User-agent: AliveAdvisorCrawler/1.7.1
Disallow: /
User-agent: AliveAdvisorCrawler/1.7.1 (whole web crawler; http://search.aliveadvisor.com; crawler@aliveadvisor.com)
Disallow: /
User-agent: madaali.de
Disallow: /
User-agent: memoryBot
Disallow: /
User-agent: whole web crawler
Disallow: /
User-agent: MegaIndex.ru/2.0
Disallow: /
User-agent: MegaIndex.ru/
Disallow: /
User-agent: SputnikBot
Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: wget
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: berlin-fu-cow
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: semager
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: InAGist URL Resolver
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Insitesbot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: IndeedBot
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: MegaIndex.ru/2.0
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: megaIndex.ru
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: ltx71
Disallow: /
User-agent: SputnikBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: seoscanners.net
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: meds-online24.com
Disallow: /
User-agent: seoscanners
Disallow: /
User-agent: GuzzleHttp
Disallow: /
User-agent: IDBot
Disallow: /
User-agent: PRTGCloudBot
Disallow: /
User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebLyzard Crawler
Disallow: /
User-agent: WebCopier v4.3
Disallow: /
User-agent: Vagabondo
Disallow: /
User-agent: Shim-Crawler
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: NimbleCrawler
Disallow: /
User-agent: hl_ftien_spider_v1.1
Disallow: /
User-agent: EMPAS_ROBOT
Disallow: /
User-agent: EDI/1.4.1 (Edacious & Intelligent Web Crawler)
Disallow: /
User-agent: (compatible; hetrix/1.8.0+http://www.accelobot.com)
Disallow: /
User-agent: WebReaper [info@webreaper.net]
Disallow: /
User-agent: WebReaper [webcreaper@otway.com]
Disallow: /
User-agent: Mozilla/5.0 (compatible; Vagabondo/Kliksafe; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Disallow: /
User-agent: Vagabondo/4.0 (webagent at wise-guys dot nl)
Disallow: /
User-agent: Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Disallow: /
# http://www.wise-guys.nl/
User-agent: webcrawler
Disallow: /
User-agent: ilse
Disallow: /
User-agent: Vagabondo
Disallow: /
User-agent: Bilbo
Disallow: /
User-agent: PiplBot
Disallow: /
User-agent: lwp-trivial/5.810
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: discoverybot
Disallow: /
User-agent: SimplePie
Disallow: /
User-agent: SimplePie/1.1.1
Disallow: /
User-agent: SimplePie/1.0.1
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Ocelli/1.4
Disallow: /
User-agent: Ocelli
Disallow: /
User-agent: ezooms.bot@gmail.com
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Ezooms/1.0
Disallow: /
User-agent: Ezooms/1.0; ezooms.bot@gmail.com
Disallow: /
#ad Crawler: http://www.a6corp.com/a6-web-scraping-policy/
User-agent: A6-Indexer/1.0
Disallow: /
#ad Crawler: http://www.grapeshot.co.uk/crawler.php
User-agent: grapeshot
Disallow: /
# http://www.proximic.com/info/spider.php
User-agent: proximic
Disallow: /
# http://www.jobdigger.nl/spider/
User-agent: JobdiggerSpider
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: rogerbot
Disallow:/
User-agent: SEOkicks-Robot
Disallow:/
User-agent: SMTBot
Disallow:/
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: EnaBot/1.2 (http://www.enaball.com/crawler.html)
Disallow: /
User-agent: EnaBot
Disallow: /
User-agent: Go-http-client/1.1
Disallow: /
User-agent: SMTBot/1.0
Disallow: /
User-agent: e-SocietyRobot
Disallow: /
User-agent: Synthesio
Disallow: /
User-agent: Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Disallow: /
User-agent: FreeFind
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Caliperbot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: SimplePie
Disallow: /
User-agent: SimplePie/1.1.3 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20081219
Disallow: /
User-Agent: trendictionbot
Disallow: /
# http://sentibot.eu/
User-agent: SentiBot
Disallow: /
User-agent: SentiBot
Disallow: /
User-agent: omgili/0.5 +http://omgili.com
Disallow: /
User-agent: Mediatoolkitbot (complaints@mediatoolkit.com)
Disallow: /
User-agent: Mediatoolkitbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: GrapeshotCrawler
Disallow:/
User-agent: Grapeshot Crawler
Disallow:/
User-agent: Grapeshot
Disallow:/
User-agent: omgili/0.5 +http://omgili.com
Disallow: /
User-agent: omgili
Disallow: /
User-agent: musobot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SimplePie
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: Mozilla/5.0 ResearchBot/0.6.2 (project: flattr-id-meta-tokens) This/1
Disallow: /
User-agent: ResearchBot/0.6.2
Disallow: /
User-agent: ResearchBot
Disallow: /
User-agent: jobs.de-Robot
Disallow: /
User-agent: FreeFind
Disallow: /
User-agent: xingbot
Disallow: /
User-agent: xing bot
Disallow: /
User-agent: xing
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: ChinasoSpider
Disallow: /
User-agent: yisouspider
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Caliperbot
Disallow: /
User-agent: spbot
Disallow: /
user-agent: EveryoneSocialBot
Disallow: /
user-agent: something
Disallow: /
user-agent: ABCdatos BotLink
user-agent: Acme.Spider
user-agent: Ahoy! The Homepage Finder
user-agent: Alkaline
user-agent: Anthill
user-agent: Walhello appie
user-agent: Arachnophilia
user-agent: Arale
user-agent: Araneo
user-agent: AraybOt
user-agent: ArchitextSpider
user-agent: Aretha
user-agent: ARIADNE
user-agent: arks
user-agent: AskJeeves
user-agent: ASpider (Associative Spider)
user-agent: ATN Worldwide
user-agent: Atomz.com Search Robot
user-agent: AURESYS
user-agent: BackRub
user-agent: Bay Spider
user-agent: Big Brother
user-agent: Bjaaland
user-agent: BlackWidow
user-agent: Die Blinde Kuh
user-agent: Bloodhound
user-agent: Borg-Bot
user-agent: BoxSeaBot
user-agent: bright.net caching robot
user-agent: BSpider
user-agent: CACTVS Chemistry Spider
user-agent: Calif
user-agent: Cassandra
user-agent: Digimarc Marcspider/CGI
user-agent: Checkbot
user-agent: ChristCrawler.com
user-agent: churl
user-agent: cIeNcIaFiCcIoN.nEt
user-agent: CMC/0.01
user-agent: Collective
user-agent: Combine System
user-agent: Conceptbot
user-agent: ConfuzzledBot
user-agent: CoolBot
user-agent: Web Core / Roots
user-agent: XYLEME Robot
user-agent: Internet Cruiser Robot
user-agent: Cusco
user-agent: CyberSpyder Link Test
user-agent: CydralSpider
user-agent: Desert Realm Spider
user-agent: DeWeb(c) Katalog/Index
user-agent: DienstSpider
user-agent: Digger
user-agent: Digital Integrity Robot
user-agent: Direct Hit Grabber
user-agent: DNAbot
user-agent: DownLoad Express
user-agent: DragonBot
user-agent: DWCP (Dridus' Web Cataloging Project)
user-agent: e-collector
user-agent: EbiNess
user-agent: EIT Link Verifier Robot
user-agent: ELFINBOT
user-agent: Emacs-w3 Search Engine
user-agent: ananzi
user-agent: esculapio
user-agent: Esther
user-agent: Evliya Celebi
user-agent: FastCrawler
user-agent: Fluid Dynamics Search Engine robot
user-agent: Felix IDE
user-agent: Wild Ferret Web Hopper #1, #2, #3
user-agent: FetchRover
user-agent: fido
user-agent: Hämähäkki
user-agent: KIT-Fireball
user-agent: Fish search
user-agent: Fouineur
user-agent: Robot Francoroute
user-agent: Freecrawl
user-agent: FunnelWeb
user-agent: gammaSpider, FocusedCrawler
user-agent: gazz
user-agent: GCreep
user-agent: GetBot
user-agent: GetURL
user-agent: Golem
user-agent: Grapnel/0.01 Experiment
user-agent: Griffon
user-agent: Gromit
user-agent: Northern Light Gulliver
user-agent: Gulper Bot
user-agent: HamBot
user-agent: Harvest
user-agent: havIndex
user-agent: HI (HTML Index) Search
user-agent: Hometown Spider Pro
user-agent: ht://Dig
user-agent: HTMLgobble
user-agent: Hyper-Decontextualizer
user-agent: iajaBot
user-agent: IBM_Planetwide
user-agent: Popular Iconoclast
user-agent: Ingrid
user-agent: Imagelock
user-agent: IncyWincy
user-agent: Informant
user-agent: InfoSeek Robot 1.0
user-agent: Infoseek Sidewinder
user-agent: InfoSpiders
user-agent: Inspector Web
user-agent: IntelliAgent
user-agent: I, Robot
user-agent: Iron33
user-agent: Israeli-search
user-agent: JavaBee
user-agent: JBot Java Web Robot
user-agent: JCrawler
user-agent: Jeeves
user-agent: JoBo Java Web Robot
user-agent: Jobot
user-agent: JoeBot
user-agent: The Jubii Indexing Robot
user-agent: JumpStation
user-agent: image.kapsi.net
user-agent: Katipo
user-agent: KDD-Explorer
user-agent: Kilroy
user-agent: KO_Yappo_Robot
user-agent: LabelGrabber
user-agent: larbin
user-agent: legs
user-agent: Link Validator
user-agent: LinkScan
user-agent: LinkWalker
user-agent: Lockon
user-agent: logo.gif Crawler
user-agent: Lycos
user-agent: Mac WWWWorm
user-agent: Magpie
user-agent: marvin/infoseek
user-agent: Mattie
user-agent: MediaFox
user-agent: MerzScope
user-agent: NEC-MeshExplorer
user-agent: MindCrawler
user-agent: mnoGoSearch search engine software
user-agent: moget
user-agent: MOMspider
user-agent: Monster
user-agent: Motor
user-agent: Muncher
user-agent: Muninn
user-agent: Muscat Ferret
user-agent: Mwd.Search
user-agent: Internet Shinchakubin
user-agent: NDSpider
user-agent: Nederland.zoek
user-agent: NetCarta WebMap Engine
user-agent: NetMechanic
user-agent: NetScoop
user-agent: newscan-online
user-agent: NHSE Web Forager
user-agent: Nomad
user-agent: The NorthStar Robot
user-agent: nzexplorer
user-agent: ObjectsSearch
user-agent: Occam
user-agent: HKU WWW Octopus
user-agent: OntoSpider
user-agent: Openfind data gatherer
user-agent: Orb Search
user-agent: Pack Rat
user-agent: PageBoy
user-agent: ParaSite
user-agent: Patric
user-agent: pegasus
user-agent: The Peregrinator
user-agent: PerlCrawler 1.0
user-agent: Phantom
user-agent: PhpDig
user-agent: PiltdownMan
user-agent: Pimptrain.com's robot
user-agent: Pioneer
user-agent: html_analyzer
user-agent: Portal Juice Spider
user-agent: PGP Key Agent
user-agent: PlumtreeWebAccessor
user-agent: Poppi
User-agent: iskanie
user-agent: PortalB Spider
user-agent: psbot
user-agent: GetterroboPlus Puu
user-agent: The Python Robot
user-agent: Raven Search
user-agent: RBSE Spider
user-agent: Resume Robot
user-agent: RoadHouse Crawling System
user-agent: RixBot
user-agent: Road Runner: The ImageScape Robot
user-agent: Robbie the Robot
user-agent: ComputingSite Robi/1.0
user-agent: RoboCrawl Spider
user-agent: RoboFox
user-agent: Robozilla
user-agent: Roverbot
user-agent: RuLeS
user-agent: SafetyNet Robot
user-agent: Scooter
user-agent: Sleek
user-agent: Search.Aus-AU.COM
user-agent: SearchProcess
user-agent: Senrigan
user-agent: SG-Scout
user-agent: ShagSeeker
user-agent: Shai'Hulud
user-agent: Sift
user-agent: Simmany Robot Ver1.0
user-agent: Site Valet
user-agent: Open Text Index Robot
user-agent: SiteTech-Rover
user-agent: Skymob.com
user-agent: SLCrawler
user-agent: Inktomi Slurp
user-agent: Smart Spider
user-agent: Snooper
user-agent: Solbot
user-agent: Spanner
user-agent: Speedy Spider
user-agent: spider_monkey
user-agent: SpiderBot
user-agent: Spiderline Crawler
user-agent: SpiderMan
user-agent: SpiderView(tm)
user-agent: Spry Wizard Robot
user-agent: Site Searcher
user-agent: Suke
user-agent: suntek search engine
user-agent: Sven
user-agent: Sygol
user-agent: TACH Black Widow
user-agent: Tarantula
user-agent: tarspider
user-agent: Tcl W3 Robot
user-agent: TechBOT
user-agent: Templeton
user-agent: TeomaTechnologies
user-agent: TITAN
user-agent: TitIn
user-agent: The TkWWW Robot
user-agent: TLSpider
user-agent: UCSD Crawl
user-agent: UdmSearch
user-agent: UptimeBot
user-agent: URL Check
user-agent: URL Spider Pro
user-agent: Valkyrie
user-agent: Verticrawl
user-agent: Victoria
user-agent: vision-search
user-agent: void-bot
user-agent: Voyager
user-agent: VWbot
user-agent: The NWI Robot
user-agent: W3M2
user-agent: WallPaper (alias crawlpaper)
user-agent: the World Wide Web Wanderer
user-agent: w@pSpider by wap4.com
user-agent: WebBandit Web Spider
user-agent: WebCatcher
user-agent: WebCopy
user-agent: webfetcher
user-agent: The Webfoot Robot
user-agent: Webinator
user-agent: weblayers
user-agent: WebLinker
user-agent: WebMirror
user-agent: The Web Moose
user-agent: WebQuest
user-agent: Digimarc MarcSpider
user-agent: WebReaper
user-agent: webs
user-agent: Websnarf
user-agent: WebSpider
user-agent: WebVac
user-agent: webwalk
user-agent: WebWalker
user-agent: WebWatch
user-agent: Wget
user-agent: whatUseek Winona
user-agent: WhoWhere Robot
user-agent: Wired Digital
user-agent: Weblog Monitor
user-agent: w3mir
user-agent: WebStolperer
user-agent: The Web Wombat
user-agent: The World Wide Web Worm
user-agent: WWWC Ver 0.2.5
user-agent: WebZinger
user-agent: XGET
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: Meltwater
Disallow: /
User-agent: Digimind
Disallow: /
User-agent: Knowings
Disallow: /
User-agent: kbcrawl
Disallow: /
User-agent: AmiSoftware
Disallow: /
User-agent: Ask n read
Disallow: /
User-agent: Qwam content intelligence
Disallow: /
User-agent: Zite
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Youmag
Disallow: /
User-agent: Synthesio
Disallow: /
User-agent: trendybuzz
Disallow: /
User-agent: spotter
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: 5emeRue
Disallow: /
User-agent: Augure
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: Corporama
Disallow: /
User-agent: readability.com
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: libwww
Disallow: /
User-agent: 5erue
Disallow: /
User-agent: adequat
Disallow: /
User-agent: adequat-systems
Disallow: /
User-agent: auramundi
Disallow: /
User-agent: coexel
Disallow: /
User-agent: leadbox
Disallow: /
User-agent: Moreover
Disallow: /
User-agent: mytwip
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: opinion-tracker
Disallow: /
User-agent: proxem
Disallow: /
User-agent: score3
Disallow: /
User-agent: trendeo
Disallow: /
User-agent: vecteurplus
Disallow: /
User-agent: verticalsearch
Disallow: /
User-agent: vsw
Disallow: /
User-agent: winello
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: infoseek
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: asknread.com
Disallow: /
User-agent: ellisphere
Disallow: /
User-agent: spotter
Disallow: /
User-agent: 008
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: AlvinetSpider
Disallow: /
User-agent: Antenne Hatena
Disallow: /
User-agent: ApocalXExplorerBot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: BizInformation
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: DISCo Pump 3.1
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: DotBot/1.1
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTrack 3.0
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: Igentia
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: InfoPath
Disallow: /
User-agent: InfoPath.3
Disallow: /
User-agent: Cyberwatcher
Disallow: /
User-agent: SafeDNSBot (https://www.safedns.com/searchbot)
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: Naverbot
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetAttache
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: OpenindexSpider
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RMA
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SightupBot
Disallow: /
User-agent: SiteBot
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: SiteSucker
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: sosospider
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: suggybot
Disallow: /
User-agent: SuperBot
Disallow: /
User-agent: SuperBot/2.6
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: TightTwatBot
Disallow: /
User-agent: Titan
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: TosCrawler
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: True_Robot/1.0
Disallow: /
User-agent: turingos
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: UrlPouls
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: VCI
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebBandit/3.50
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: webcopy
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: webmirror
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: website extractor
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebStripper/2.02
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: wget
Disallow: /
User-agent: WikioFeedBot
Disallow: /
User-agent: WinHTTrack
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: yacy
Disallow: /
User-agent: YRSPider
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zookabot
Disallow: /
## EXCLUSION BOTS
User-agent: ConveraCrawler
User-agent: Jetbot
User-agent: Meltwater
User-agent: Digimind
User-agent: Knowings
User-agent: kbcrawl
User-agent: AmiSoftware
User-agent: Ask n read
User-agent: Qwam content intelligence
User-agent: Zite
User-agent: FlipboardProxy
User-agent: Youmag
User-agent: Synthesio
User-agent: trendybuzz
User-agent: spotter
User-agent: linkfluence
User-agent: 5emeRue
User-agent: Augure
User-agent: Sindup
User-agent: Corporama
User-agent: readability.com
User-agent: grub-client
User-agent: ia_archiver
User-agent: ia_archiver-web.archive.org
User-agent: k2spider
User-agent: libwww
User-agent: 5erue
User-agent: adequat
User-agent: adequat-systems
User-agent: auramundi
User-agent: coexel
User-agent: leadbox
User-agent: Moreover
User-agent: mytwip
User-agent: NewsNow
User-agent: Newzbin
User-agent: opinion-tracker
User-agent: proxem
User-agent: score3
User-agent: trendeo
User-agent: vecteurplus
User-agent: verticalsearch
User-agent: vsw
User-agent: winello
User-agent: Fetch
User-agent: infoseek
User-agent: MSIECrawler
User-agent: sitecheck.internetseer.com
User-agent: TeleportPro
User-agent: Zealbot
User-agent: asknread.com
User-agent: ellisphere
User-agent: spotter
User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: yacy
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Linguee Bot
Disallow: /
User-agent: BLP_bbot/0.1
Disallow: /
User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: YandexSomething/1.0
Disallow: /
User-agent: Java/1.6.0_04
Disallow: /
User-agent: 8484 Boston Project v 1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; m)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; H)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; P)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; I)
Disallow: /
User-agent: +BecomeBot/3.0
Disallow: /
User-agent: AISearchBot
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: AmazonCloudServices
Disallow: /
User-agent: Apexoo Spider 1.0
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: Arachnoidea
Disallow: /
User-agent: ArchitextSpider
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: DotBot/1.1
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
User-agent: SemrushBot-SA/0.97
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Disallow: /
User-agent: AsapBot
Disallow: /
User-agent: semantic-visions.com crawler
Disallow: /
User-agent: MetaURI API/2.0 +metauri.com
Disallow: /
User-agent: MetaURI
Disallow: /
User-agent: citeseerxbot
Disallow: /
User-agent: autokrawl
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /
User-agent: becomebot
Disallow: /
User-agent: BecomeBot/3.0
Disallow: /
User-agent: Bitacle
Disallow: /
User-agent: Bitacle bot
Disallow: /
User-agent: Bitacle bot/1.1
Disallow: /
User-agent: Bitacle*
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: BobCrawl
Disallow: /
User-agent: boitho
Disallow: /
User-agent: boitho.com
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: bot/1.0
Disallow: /
User-agent: bot/1.0+(bot;+http://;+bot@bot.bot)
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: botmobi
Disallow: /
User-agent: bsalsa
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Butterfly
Disallow: /
User-agent: Cazoodle-Bot
Disallow: /
User-agent: CazoodleBot
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: Charlotte
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: classroombot
Disallow: /
User-agent: ConveraCrawler/0.9d
Disallow: /
User-agent: ConveraCrawler/0.9e
Disallow: /
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: crawl13.exabot.com
Disallow: /
User-agent: crawl6.exabot.com
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: cyberpatrol
Disallow: /
User-agent: cyberpatrolcrawler
Disallow: /
User-agent: datacha0s/2.0
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
Disallow: /
User-agent: dcbot
Disallow: /
User-agent: del.icio.us
Disallow: /
User-agent: del.icio.us-thumbnails/1.0
Disallow: /
User-agent: DepSpid
Disallow: /
User-agent: discobot
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: DTAAgent
Disallow: /
User-agent: eagle
Disallow: /
User-agent: ee:
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: envolk
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: EUDORA
Disallow: /
User-agent: Everest-Vulcan
Disallow: /
User-agent: exabot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Exabot-Images/1.0
Disallow: /
User-agent: Exabot-Images/3.0
Disallow: /
User-agent: Exabot/2.0
Disallow: /
User-agent: Exabot/3.0
Disallow: /
User-agent: exooba
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: FairShare
Disallow: /
User-agent: fast
Disallow: /
User-agent: fast Enterprise crawler
Disallow: /
User-agent: fast metaweb crawler
Disallow: /
User-agent: FAST-WebCrawler
Disallow: /
User-agent: fast-Webcrawler
Disallow: /
User-agent: findlinks/1.1.1-a5
Disallow: /
User-agent: findlinks/1.1.3-beta9
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: flatlandbot
Disallow: /
User-agent: jack
Disallow: /
User-agent: JS-Kit URL Resolver
Disallow: /
User-agent: KomodiaBot
Disallow: /
User-agent: FollowSiteBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: FunWebProducts
Disallow: /
User-agent: FunWebProducts f39-2350
Disallow: /
User-agent: FyberSpider
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: genieBot
Disallow: /
User-agent: genieBot (wgao@genieknows.com)
Disallow: /
User-agent: genieBot+(wgao@genieknows.com)
Disallow: /
User-agent: genieBot+wgao@genieknows.com
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Gigabot*
Disallow: /
User-agent: Gigabot/2.0
Disallow: /
User-agent: GingerCrawler
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: HenryTheMiragoRobot
Disallow: /
User-agent: heritrix
Disallow: /
User-agent: heritrix/1.12.1 +http://www.page-store.com)
Disallow: /
User-agent: hloader
Disallow: /
User-agent: HMSE_Robot
Disallow: /
User-agent: HornySexSearch
Disallow: /
User-agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /
User-agent: http://www.cuill.com/twiceler/robot.html
Disallow: /
User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/)
Disallow: /
User-agent: httplib
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archive
Disallow: /
User-agent: ia_archiver-web.archive.org
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: IBM
Disallow: /
User-agent: IBM_Planetwide
Disallow: /
User-agent: ichiro/2.0
Disallow: /
User-agent: ichiro/2.0 (ichiro@nttr.co.jp)
Disallow: /
User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
Disallow: /
User-agent: IconSurf
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: InfoPath
Disallow: /
User-agent: InfoPath.1
Disallow: /
User-agent: InfoPath.2
Disallow: /
User-agent: IntegraTelecom
Disallow: /
User-agent: internetseer
Disallow: /
User-agent: InternetSeer.com
Disallow: /
User-agent: ips-agent
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: IRLbot/3.0
Disallow: /
User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: Isidorus
Disallow: /
User-agent: Java Bot 2.0
Disallow: /
User-agent: Java/1.5.0_05
Disallow: /
User-agent: Java/1.5.0_10
Disallow: /
User-agent: JavaBot
Disallow: /
User-agent: JavaBot 2.0
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Jyxobot
Disallow: /
User-agent: kalooga
Disallow: /
User-agent: KaloogaBot
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Disallow: /
User-agent: kyluka
Disallow: /
User-agent: LaBot
Disallow: /
User-agent: larbin
Disallow: /
User-agent: larbin_2.6.1
Disallow: /
User-agent: larbin_2.6.1+(larbin2.6.1@unspecified.mail)
Disallow: /
User-agent: laycat
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: LexiBotWeb Image Collector
Disallow: /
User-agent: libcurl-agent
Disallow: /
User-agent: LinkAider
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkWalker/2.0
Disallow: /
User-agent: LinkWalker/2.0 - www.seventwentyfour.com
Disallow: /
User-agent: LMQueueBot/0.2
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: LocalBot
Disallow: /
User-agent: LocalcomBot
Disallow: /
User-agent: looksmart
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: MantraAgent
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MJ12bot/v1.2.0
Disallow: /
User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? )
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: Modified GMMBotRightHere
Disallow: /
User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Disallow: /
User-agent: Mozilla/5.0+(compatible;+BecomeBot/3.0;++http://www.become.com/site_owners.html)
Disallow: /
User-agent: MSR-ISRCCrawler
Disallow: /
User-agent: nambu
Disallow: /
User-agent: NationalDirectory-SuperSpider
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: nexen
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: NG/2.0
Disallow: /
User-agent: nicebot
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: noxtrumbot/1.0
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: obot
Disallow: /
User-agent: Ocelli
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Offline Explorer/1.5
Disallow: /
User-agent: Offline Explorer/1.9
Disallow: /
User-agent: Offline Explorer/2.5
Disallow: /
User-agent: OmniExplorer
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: OmniExplorer_Bot/1.07
Disallow: /
User-agent: OmniExplorer_Bot/1.09
Disallow: /
User-agent: OmniExplorer_Bot/1.10
Disallow: /
User-agent: OnTownsBot
Disallow: /
User-agent: OOZBOT
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind data gathere
Disallow: /
User-agent: OpenX
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: page_prefetcher
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: Piffany_Web_Cacher_v0.91
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: psbot/0.1
Disallow: /
User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html)
Disallow: /
User-agent: PSIBots
Disallow: /
User-agent: QEAVis
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: Radiation Retriever 1.1
Disallow: /
User-agent: Rankivabot
Disallow: /
User-agent: REAP-Crawler
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-agent: RiceComputerArchitecture
Disallow: /
User-agent: RMA
Disallow: /
User-agent: robotgenius
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: ru_com_viewer
Disallow: /
User-agent: RufusBot
Disallow: /
User-agent: SapphireWebCrawler
Disallow: /
User-agent: Twiceler www.cuill.com/robots.html
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: wget
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: IsraBot
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: fast
Disallow: /
User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Disallow: /
User-agent: Twiceler www.cuill.com/twiceler/robot.html
Disallow: /
User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Disallow: /
User-agent: sbider
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: SBIder/SBIder-0.8.2-dev
Disallow: /
User-agent: scooter
Disallow: /
User-agent: Scooter/2.0
Disallow: /
User-agent: scoutjet
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: searchme
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: Sensis Web Crawler
Disallow: /
User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)
Disallow: /
User-agent: Sensis+Web+Crawler+
Disallow: /
User-agent: Servage
Disallow: /
User-agent: ServageRobot
Disallow: /
User-agent: setooz
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: Sidewinder
Disallow: /
User-agent: Sika
Disallow: /
User-agent: sitecat
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: Slicehost
Disallow: /
User-agent: Slurp.so/1.0
Disallow: /
User-agent: slurp@inktomi
Disallow: /
User-agent: slurp@inktomi.com
Disallow: /
User-agent: Slurp/2.0-KiteHourly
Disallow: /
User-agent: Slurp/2.0-OwlWeekly
Disallow: /
User-agent: Slurp/2.0j
Disallow: /
User-agent: Slurp/3.0-AU
Disallow: /
User-agent: SlySearch
Disallow: /
User-agent: SlySearch/1.0
Disallow: /
User-agent: SlySearch/1.x
Disallow: /
User-agent: Snapbot
Disallow: /
User-agent: Snapbot/1.0
Disallow: /
User-agent: SnapPreviewBot
Disallow: /
User-agent: Snappy
Disallow: /
User-agent: Snappybot
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: sogou+spider
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: speedyspider
Disallow: /
User-agent: sphsearch
Disallow: /
User-agent: sproose
Disallow: /
User-agent: StackRambler/2.0
Disallow: /
User-agent: StackRambler/2.0+(MSIE+incompatible)
Disallow: /
User-agent: SuperBot/4.4.0.60 (Windows XP)
Disallow: /
User-agent: SuperPages
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: SurveyBot/2.3
Disallow: /
User-agent: SurveyBot/2.3 (Whois Source)
Disallow: /
User-agent: uberbot
Disallow: /
User-agent: UltraSeek
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
Disallow: /
User-agent: ViolaBot
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebCopier v3.2a
Disallow: /
User-agent: WebCopier v3.3
Disallow: /
User-agent: WebCopier v3.3.0
Disallow: /
User-agent: WebCopier v3.3.2
Disallow: /
User-agent: WebCrawler
Disallow: /
User-agent: WebDataCentreBot
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebmasterWorld Extractor
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: Websense
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webspider
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebViewer
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: WebZIP/3.5 (http://www.spidersoft.com)
Disallow: /
User-agent: whois.de
Disallow: /
User-agent: whoisde.de
Disallow: /
User-agent: WillyBot
Disallow: /
User-agent: Win32
Disallow: /
User-agent: worio
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Xenu Link Sleuth 1.2i
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: Xenu Link Sleuth/1.3.8
Disallow: /
User-agent: Y!J
Disallow: /
User-agent: y!j-bsc
Disallow: /
User-agent: Y!J-BSC
Disallow: /
User-agent: Y!J-BSC/1.0
Disallow: /
User-agent: TunitinBot
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: Meltwater
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Digimind
Disallow: /
User-agent: Knowings D
Disallow: /
User-agent: kbcrawl
Disallow: /
User-agent: Wget
Disallow: /
User-agent: AmiSoftware
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: Ask n read
Disallow: /
User-agent: Qwam content intelligence
Disallow: /
User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Disallow: /
User-agent: y!j/1.0
Disallow: /
User-agent: Yahoo-Blogs
Disallow: /
User-agent: Yahoo-Blogs/v3.9
Disallow: /
User-agent: Yahoo-MMCrawler/3.x
Disallow: /
User-agent: YahooSeeker
Disallow: /
User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
Disallow: /
User-agent: YahooYSMcm
Disallow: /
User-agent: Zite
Disallow: /
User-agent: flipboard
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Youmag
Disallow: /
User-agent: YahooYSMcm/2.0.0
Disallow: /
User-agent: yandex
Disallow: /
User-agent: YandexSomething
Disallow: /
User-agent: Yanga
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: yoogliFetchAgent/0.1
Disallow: /
User-agent: Zao
Disallow: /
User-agent: zermelo
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: zibber
Disallow: /
User-agent: zibber v0.1
Disallow: /
User-agent: zibber-v0.1(www.zibb.com/crawler/)
Disallow: /
User-agent: ZupeeCrawler
Disallow: /
User-agent: zyborg
Disallow: /
User-agent: ZyBorg/1.0
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Snoothbot
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Daum
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: WeSEE
Disallow: /
User-agent: ShopWiki
Disallow: /
user-agent: crystalsemantics
disallow: /
user-agent: f-six
disallow: /
user-agent: sitebot
disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: scrapybot
Disallow: /
User-agent: VocusBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: ezooms.bot
Disallow: /
User-agent: skimlinks.com
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: wikiwix-bot
Disallow: /
User-agent: semantissimo
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: LemonSources
Disallow: /
User-agent: discobot
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: ltbot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: language-tools.com
Disallow: /
User-agent: TREC-KBA-Bot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: Socialradarbot
Disallow: /
User-agent: Havij
Disallow: /
User-agent: proximic
Disallow: /
User-agent: MireoBot
Disallow: /
User-agent: brandwatch
Disallow: /
User-agent: seoprofiler
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Bot clicker
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: EC2LinkFinder
Disallow: /
User-agent: ExaleadCloudview
Disallow: /
User-agent: AdMedia
Disallow: /
User-agent: grepnetstat.com
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: NerdByNature
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: netseer
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: netseer.com
Disallow: /
User-agent: nagios
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: FeedRetriever
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: Argus Presse
Disallow: /
User-agent: IBM ICA Crawler
Disallow: /
User-agent: nutch agent
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: PaperLiBot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: ip-web-crawler.com
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: LinksCrawler
Disallow: /
User-agent: archive.org
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: netEstate
Disallow: /
User-agent: MyNutchTest
Disallow: /
User-agent: admantx
Disallow: /
User-agent: genieo.com
Disallow: /
User-agent: mixdata.com
Disallow: /
User-agent: ubermetrics-technologies
Disallow: /
User-agent: grapeshot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: WASALive
Disallow: /
User-agent: URLAppendBot
Disallow: /
User-agent: MaxPointCrawler
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Aboundexbot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: psbot
Disallow: /
User-Agent: Slurp
User-agent: Yahoo! Slurp
User-agent: Baiduspider
User-agent: TweetedTimes Bot
User-agent: TweetmemeBot
User-agent: PaperLiBot
User-agent: BomboraBot
User-agent: WF search
User-agent: heritrix
User-agent: expo9
User-agent: NetSeer
User-agent: AddThis.com
User-agent: Feed
User-agent: Sogou
User-agent: ZumBot
User-agent: Feedspot
User-agent: acebookexternalhit
User-agent: StorygizeBot
User-agent: Scrapy
User-agent: GermCrawler
User-agent: Netvibes
User-agent: Vagabondo
User-agent: Genieo
User-agent: um-IC
User-agent: Typhoeus
User-agent: newspaper
User-agent: weborama-fetcher
User-agent: ScooperBot
User-agent: BYO-2
User-agent: SimplePie
User-agent: GroupHigh
User-agent: feedparser
User-agent: UniversalFeedParser
User-agent: Pulsepoint
User-agent: Steeler
User-agent: GarlikCrawler
User-agent: Flipboard
User-agent: FlipboardRSS
User-agent: Moreover
User-agent: Mercury
User-agent: FeedBurner
User-agent: Yahoo
User-agent: omgili
User-agent: Bloglovin
User-agent: YahooCacheSystem
User-agent: mTalk
User-agent: EveryoneSocialBot
User-agent: EzineArticlesLinkScanner
User-agent: dlvr.it
User-agent: pfblogs
User-agent: Aol
User-agent: ToutiaoSpider
User-agent: Photon
User-agent: PicoFeed
User-agent: um-FC
User-agent: Digg
User-agent: Synaptive
User-agent: YisouSpider
User-agent: BYO-4
User-agent: Feedbin
User-agent: Twingly
User-agent: Tiny
User-agent: WWW-Mechanize
User-agent: CommaFeed
User-agent: TkBot
User-agent: TinEye-bot
User-agent: Exabot
User-agent: coccoc
User-agent: BLEXBot
User-agent: uMBot-LN
User-agent: QuerySeekerSpider
User-agent: Yandex
User-agent: YandexBot
User-agent: YandexMobileBot
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: sogou spider
User-agent: Sogou web spider
User-agent: Baiduspider
User-agent: YoudaoBot
User-agent: MJ12bot
User-agent: spbot
User-agent: 360Spider
User-agent: SeznamBot
User-Agent: AhrefsBot
User-Agent: Linkdex
User-agent: linkdexbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: CFNetwork
User-agent: SputnikBot
User-agent: TurnitinBot
User-agent: proximic
User-agent: grapeshot
User-agent: VigLink
User-agent: Clickagy
User-agent: YaK
User-agent: DeuSu
User-agent: Taboolabot
User-agent: Pinterestbot
User-agent: psbot
User-agent: Plukkie
User-agent: GetIntent
User-agent: Facebot
User-agent: linkdexbot
User-agent: Vegi
User-agent: istellabot
User-agent: Leikibot
User-agent: Claritybot
User-agent: PiplBot
User-agent: Feedspotbot
User-agent: moatbot
User-agent: Yeti
User-agent: contxbot
User-agent: SEOkicks-Robot
User-agent: rogerbot
User-agent: Discordbot
User-agent: Tapatalk
User-agent: Mail.RU_Bot
User-agent: coccocbot
User-agent: HybridBot
User-agent: Slack-ImgProxy
User-agent: adbeat_bot
User-agent: Kraken
User-agent: MojeekBot
User-agent: calculon
User-agent: mcsbot
User-agent: avkzarabotok
User-agent: SearchmetricsBot
User-agent: Uptimebot
User-agent: yacybot
User-agent: Applebot
User-agent: Gigabot
User-agent: g2reader-bot
User-agent: Seoterritory
User-agent: AhrefsBot
User-agent: feedbot
User-agent: Findxbot
User-agent: Wotbox
User-agent: SMTBot
User-agent: Ocarinabot
User-agent: Dragonbot
User-agent: Zombiebot
User-agent: metabot.ru
User-agent: RSSMicro
User-agent: _zbot
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Cliqzbot
User-agent: DotBot
User-agent: Exabot
User-agent: moatbot
User-agent: Tapatalk
Disallow: /
User-agent: GingerCrawler
Disallow: /
User-agent: nutch
Disallow: /
User-agent: Urlbot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: 008
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Speedy Spider
Disallow: /
User-agent: Accelobot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: discobot
Disallow: /
User-agent: AboutUsBot
Disallow: /
User-agent: Vocus
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: http://ahrefs.com/robot
Disallow: /
User-agent: http://www.genieo.com/webfilter.html
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: linkdex.com/v2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Disallow: /
User-agent: NetcraftSurveyAgent/1.0; +info@netcraft.com
Disallow: /
User-agent: NetcraftSurveyAgent/1.0
Disallow: /
User-agent: NetcraftSurveyAgent
Disallow: /
User-agent: NetcraftSurveyAgent/2.0
Disallow: /
User-agent: SemrushBot/0.95
Disallow: /
User-agent: MJ12bot/v1.4.3
Disallow: /
User-agent: Java/1.6.0_04
Disallow: /
User-agent: JS-Kit URL Resolver, http://js-kit.com/
Disallow: /
User-agent: JS-Kit
Disallow: /
User-agent: NetShelter ContentScan
Disallow: /
User-agent: Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Disallow: /
User-agent: Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Disallow: /
User-agent: Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Disallow: /
User-agent: YYSpider
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: PagePeeker2
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: Abonti
Disallow: /
User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: Image\ Sucker
Disallow: /
User-agent: SuperBot
Disallow: /
User-agent: Web\ Image\ Collector
Disallow: /
User-agent: Web\ Sucker
Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: linko
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
Crawl-delay: 10
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
ФРГ - 188.40.49.136
Hetzner Online AG
Hetzner Online GmbH
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 09 Jan 2020 11:11:13 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Link: ; rel=preload; as=script, ; rel=preload; as=style, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=style, ; rel=preload; as=script, ; rel=preload; as=script, ; rel=preload; as=script, ; rel=preload; as=style, ; rel=preload; as=script, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=image, ; rel=preload; as=style
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: c258e8ae2af52fe728617eff7aa33111=dgp3m0dmanb795t2dpo3b6hft3; path=/
Last-Modified: Thu, 09 Jan 2020 11:11:13 GMT
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: no-referrer-when-downgrade
Content-Security-Policy: default-src * data: 'unsafe-eval' 'unsafe-inline'
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Кнопка для анализа сайта в один клик, для установки перетащите ссылку на "Панель закладок"