Возраст домена | n/a |
Дата окончания | n/a |
PR | 5 |
ИКС | n/a |
Страниц в Google | 1490000 |
Страниц в Яндексе | 1 |
Dmoz | Нет |
Яндекс Каталог | Нет |
Alexa Traffic Rank | 38117 |
Alexa Country | ![]() |
История изменения показателей | Авторизация |
Идет сбор информации... Обновить
Rätsel Hilfe - die große Kreuzworträtsel Hilfe mit über 1 Million Rätsellösungen
n/a
Rätsel Hilfe 𝐰𝐰𝐰.𝐫𝐚𝐞𝐭𝐬𝐞𝐥-𝐡𝐢𝐥𝐟𝐞.𝐝𝐞 Lexikon seit über 10 Jahren ✔ Alle Schwedenrätsel & Kreuzworträtsel Lösungen ✔ Kreuzworträtsel kostenlos online lösen ✔
UTF-8
67.19 КБ
412
3 548 симв.
3 050 симв.
Данные предоставлены сервисом semrush
Сайт | Общие фразы | PR | тИЦ | Alexa Rank | Alexa Country | |
---|---|---|---|---|---|---|
![]() ![]() |
1 | 7 |
180 | 18033 | ![]() | |
![]() |
1 | 4 |
0 | 2565964 | ![]() |
|
![]() ![]() |
1 | 6 |
10 | 5475843 | ![]() |
|
![]() ![]() |
1 | 5 |
0 | 28580 | ![]() | |
![]() |
1 | n/a | 0 | 1816253 | ![]() |
|
![]() |
1 | n/a | 0 | 1152748 | ![]() | |
![]() |
1 | n/a | 0 | Нет данных | ![]() |
|
![]() |
1 | n/a | 0 | 1512289 | ![]() | |
![]() |
1 | n/a | 0 | 1819596 | ![]() | |
![]() |
1 | 2 |
0 | 13806060 | ![]() |
|
Еще 40 сайтов после авторизации |
Данные предоставлены сервисом semrush
Счетчик | Посетители за 24 часа | Просмотры | Просмотров на посетителя |
---|---|---|---|
![]() | Нет доступа | Нет доступа | n/a |
Domain: raetsel-hilfe.de
Nserver: ns1.routing.net
Nserver: ns2.routing.net
Nserver: ns3.routing.net
Status: connect
Changed: 2016-06-29T15:28:40+02:00
[Tech-C]
Type: PERSON
Name: Ronny Woick
Organisation: R. Woick
Address: Goepelstraße 73-75
PostalCode: 15234
City: Frankfurt a.d. Oder
CountryCode: DE
Phone: +49 335 28463355
Email: cg.soft@yahoo.de
Changed: 2016-03-18T08:58:42+01:00
[Zone-C]
Type: PERSON
Name: Ronny Woick
Organisation: R. Woick
Address: Goepelstraße 73-75
PostalCode: 15234
City: Frankfurt a.d. Oder
CountryCode: DE
Phone: +49 335 28463355
Email: cg.soft@yahoo.de
Changed: 2016-03-18T08:58:42+01:00
User-agent: *
Allow: /*?v*
Allow: /*?client*
Allow: /*collect?*
Disallow: /*?
Disallow: /*?*
Disallow: /?
# Disallow: /.js$
# Disallow: /*.js$
# Disallow: /*.css$
Disallow: /.php$
Disallow: /*.php
Disallow: /*.php?*
Disallow: /*.php$
Disallow: /*.sql$
Disallow: /*.tgz$
Disallow: /*.bak$
Disallow: /*.log$
Disallow: /*.txt$
Disallow: /*?SID=
Disallow: /*?SID=*
Disallow: /?SID=
Disallow: /*?sid=
Disallow: /*?sid=*
Disallow: /?sid=
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /privatenachrichten/
Disallow: /errorlog/
Disallow: /cache/
Disallow: /inc/
Disallow: /log/
Disallow: /upload/
Disallow: /admin/
Disallow: /rechtschreib/
Disallow: /template/content/
Disallow: /template/emoji/
Disallow: /template/load/
Disallow: /template/design/
Disallow: /template/werbung/
Sitemap: https://www.raetsel-hilfe.de/raetsel-blog/sitemap.xml
Sitemap: https://www.raetsel-hilfe.de/raetsel-seiten/sitemap.xml
Sitemap: https://www.raetsel-hilfe.de/raetsel-news/sitemap.xml
User-agent: OrangeBot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-Agent: Linkdexbot/2.0
Disallow: /
User-Agent: Linkdexbot/2.1
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-agent: Sogou web spider Sogou web spider/4.0
Disallow: /
User-agent: sogou web spider
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: Opoint
Disallow: /
User-agent: 008
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: ClarityDailyBot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; ClarityDailyBot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
Disallow: /
User-agent: PAD-bot/9.0
Disallow: /
User-agent: PAD-bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)
Disallow: /
User-agent: SoftListBot/2.2
Disallow: /
User-agent: SoftListBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: NativeHost
Disallow: /
User-agent: newsme/1.0; feedback@news.me
Disallow: /
User-agent: newsme/1.0
Disallow: /
User-agent: newsme
Disallow: /
User-agent: Niki-Bot
Disallow: /
User-agent: NING/1.0
Disallow: /
User-agent: NING
Disallow: /
User-agent: peerindex
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Disallow: /
User-agent: Diffbot/0.1
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net)
Disallow: /
User-agent: elefent
Disallow: /
User-agent: elefent/1.2
Disallow: /
User-agent: crawler.elefent.net
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 4.0
Disallow: /
User-agent: MS Search 5.0
Disallow: /
User-agent: MS Search 6.0
Disallow: /
User-agent: MS Search 7.0
Disallow: /
User-agent: MS Search
Disallow: /
User-agent: CCBot/2.0 (http://commoncrawl.org/faq/)
Disallow: /
User-agent: CCBot/2.0
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
Disallow: /
User-agent: Mail.RU_Bot/Robots/2.0
Disallow: /
User-agent: AOL_Daniel_Clark_Spider/Nutch-0.9 (AOL Search; danielaclark1@aol.com)
Disallow: /
User-agent: AolOnScraper/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html; gzip)
Disallow: /
User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /
User-agent: Apache Sites Search Facet
Disallow: /
User-agent: bigdata
Disallow: /
User-agent: Fast corporate crawler
Disallow: /
User-agent: 008
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: magpie-crawler/1.1
Disallow: /
User-agent: MagpieRSS
Disallow: /
User-agent: psnsearch
Disallow: /
User-agent: SLCC2
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: GuzzleHttp
Disallow: /
User-agent: Go 1.1 package http
Disallow: /
User-agent: Go 1.1
Disallow: /
User-agent: Go 1.2
Disallow: /
User-agent: Go 1.2 package http
Disallow: /
User-agent: gocrawl v0.4
Disallow: /
User-agent: gocrawl
Disallow: /
# User-agent: Google-HTTP-Java-Client
# Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: Hatena Star
Disallow: /
User-agent: PagePeeker.com
Disallow: /
User-agent: PagePeeker
Disallow: /
User-agent: _agentname
Disallow: /
User-agent: agentname
Disallow: /
User-agent: 1 in Business Networking, www.MarcusWalker.ws
Disallow: /
User-agent: Business Networking
Disallow: /
User-agent: AntiOpensourceBot/1.1
Disallow: /
User-agent: AntiOpensourceBot
Disallow: /
User-agent: antibot-V1.1.5/i586-linux-2.2
Disallow: /
User-agent: Anti-XBOCT/2007 [xxx] (OS; X; JU)
Disallow: /
User-agent: AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Disallow: /
User-agent: Anti-XBOCT
Disallow: /
User-agent: AntBot
Disallow: /
User-agent: Ant-Nutch
Disallow: /
User-agent: AntBot/1.0
Disallow: /
User-agent: AnzwersCrawl/2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: Wotbox/2.01 (+http://www.wotbox.com/bot/)
Disallow: /
User-agent: Wotbox
Disallow: /
User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPicker /1.0
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-Agent: litefinder
Disallow: /
User-agent: Friendica
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Flamingo
Disallow: /
User-agent: Faraday v0.8.8
Disallow: /
User-agent: Faraday
Disallow: /
User-agent: Barkrowler 0.1.6
Disallow: /
User-agent: Barkrowler
Disallow: /
User-Agent: ExperiBot_v1
Disallow: /
User-Agent: ExperiBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-Agent: omgilibot
Disallow: /
User-Agent: omgili
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: hloader
Disallow: /
User-agent: ConveraCrawler/0.9e
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: YahooFeedSeeker/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://publisher.yahoo.com/rssguide)
Disallow: /
User-agent: YahooFeedSeeker/2.0+
Disallow: /
User-agent: YahooFeedSeeker
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Disallow: /
User-agent: ExtLinksBot/1.5
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: NCBot
Disallow: /
User-agent: Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html)
Disallow: /
User-agent: Testbot/1.0
Disallow: /
User-agent: Testbot
Disallow: /
User-agent: Bot !Susie
Disallow: /
User-agent: Slurp Disallow: /
User-agent: !Susie
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Disallow: /
User-agent: GroupHigh/1.0
Disallow: /
User-agent: G-i-g-a-b-o-t
Disallow: /
User-agent: SeznamBot/3.2
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Disallow: /
User-agent: Helloo/1.0.4
Disallow: /
User-agent: Mozilla/5.0 (compatible; Helloo/1.0.4; +http://www.helloo.io/;)
Disallow: /
User-agent: SiteExplorer/1.1b
Disallow: /
User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /
User-agent: Mail.RU_Bot/Robots
Disallow: /
User-agent: attach
Disallow: /
User-agent: ASPSeek
Disallow: /
User-agent: appie
Disallow: /
User-agent: AbachoBOT
Disallow: /
User-agent: autoemailspider
Disallow: /
User-agent: anarchie
Disallow: /
User-agent: antibot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: B2w
Disallow: /
User-agent: BackWeb
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: Bandit
Disallow: /
User-agent: BatchFTP
Disallow: /
User-agent: Black\ Hole
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bot\ mailto
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: Buddy
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: bumblebee
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: ClariaBot
Disallow: /
User-agent: curl
Disallow: /
User-agent: clsHTTP
Disallow: /
User-agent: ChinaClaw
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: Collector
Disallow: /
User-agent: COAST\ WebMaster
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: ColdFusion
Disallow: /
User-agent: Copier
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DA
Disallow: /
User-agent: DTS\ Agent
Disallow: /
User-agent: DISCo\ Pump
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Diamond
Disallow: /
User-agent: Download\ Demon
Disallow: /
User-agent: Download\ Wonder
Disallow: /
User-agent: Downloader
Disallow: /
User-agent: dloader
Disallow: /
User-agent: Drip
Disallow: /
User-agent: eCatch
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: Express\ WebPictures
Disallow: /
User-agent: Extreme\ Picture\ Finder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EasyDL
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: EyeNetIE
Disallow: /
User-agent: FAST\ WebCrawler
Disallow: /
User-agent: masscan/1.0 (https://github.com/robertdavidgraham/masscan)
Disallow: /
User-agent: masscan/1.0
Disallow: /
User-agent: masscan
Disallow: /
User-agent: semantic-visions.com crawler
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: ADmantX Platform Semantic Analyzer
Disallow: /
User-agent: proximic
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: GetintentCrawler
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: zgrab
Disallow: /
User-agent: Pigafetta
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: postano
Disallow: /
User-agent: Prlog
Disallow: /
User-agent: PulseCrawler/1.1
Disallow: /
User-agent: QuerySeekerSpider
Disallow: /
User-agent: quipu
Disallow: /
User-agent: Raven
Disallow: /
User-agent: RavenCrawler
Disallow: /
User-agent: RBot
Disallow: /
User-agent: Recorded Future
Disallow: /
User-agent: Riddler
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: RukiCrawler
Disallow: /
User-agent: SafeDNSBot
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Searchie
Disallow: /
User-agent: semantic-visions.com crawler
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: seocharger-robot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SEOlyticsCrawler
Disallow: /
User-agent: seoscanners
Disallow: /
User-agent: seoscanners.net
Disallow: /
User-agent: ShowyouBot (http://showyou.com/crawler)
Disallow: /
User-agent: simplecrawler
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-Agent: siteluxbot
Disallow: /
User-agent: SmeshBot
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: srmse/Nutch
Disallow: /
User-agent: SoftListBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: squirrly
Disallow: /
User-agent: ssearch_bot
Disallow: /
User-agent: Stratagems Kumo
Disallow: /
User-agent: tbot-nutch
Disallow: /
User-agent: TWMBot
Disallow: /
User-agent: Typhoeus - https://github.com/typhoeus/typhoeus
Disallow: /
User-agent: uipbot
Disallow: /
User-agent: uMBot-LN
Disallow: /
User-agent: UptimeRobot/2.0
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: WebIndex
Disallow: /
User-agent: VeBot
Disallow: /
User-agent: VeriCiteCrawler
Disallow: /
User-agent: WeSEE_Bot
Disallow: /
User-agent: WiseGuys Robot
Disallow: /
User-agent: voltron
Disallow: /
User-agent: woobot
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: X-CAD-SE
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: Yahoo Pipes 1.0
Disallow: /
User-agent: YioopBot
Disallow: /
User-agent: crawler.vs.uni-duisburg-essen.de
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: FileHound
Disallow: /
User-agent: Fetch\ API\ Request
Disallow: /
User-agent: FlashGet
Disallow: /
User-agent: FlickBot
Disallow: /
User-agent: FrontPage
Disallow: /
User-agent: FreeFind.com
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: GetSmart
Disallow: /
User-agent: Generic
Disallow: /
User-agent: Go!Zilla
Disallow: /
User-agent: Go-Ahead-Got-It
Disallow: /
User-agent: gotit
Disallow: /
User-agent: Grabber
Disallow: /
User-agent: GrabNet
Disallow: /
User-agent: Grafula
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: HMView
Disallow: /
User-agent: Heretrix
Disallow: /
User-agent: HitboxDoctor
Disallow: /
User-agent: HTTPapp
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTPTrack
Disallow: /
User-agent: HTTPviewer
Disallow: /
User-agent: httplib
Disallow: /
User-agent: httpfetcher
Disallow: /
User-agent: httpscraper
Disallow: /
User-agent: hloader
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: InterGET
Disallow: /
User-agent: Internet\ Ninja
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: InternetSeer.com
Disallow: /
User-agent: Iria
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: JetCar
Disallow: /
User-agent: JOC
Disallow: /
User-agent: JOC\ Web\ Spider
Disallow: /
User-agent: JoBo
Disallow: /
User-agent: Java
Disallow: /
User-agent: JustView
Disallow: /
User-agent: Jonzilla
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin\ Spider
Disallow: /
User-agent: Keyword\ Density
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LeechFTP
Disallow: /
User-agent: Lachesis
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb
Disallow: /
User-agent: Libby_
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: lftp
Disallow: /
User-agent: likse
Disallow: /
User-agent: Link
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp\ request
Disallow: /
User-agent: Magnet
Disallow: /
User-agent: Mag-Net
Disallow: /
User-agent: Mass\ Downloader
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Missigua
Disallow: /
User-agent: Mewsoft\ Search\ Engine
Disallow: /
User-agent: moget
Disallow: /
User-agent: Mata\ Hari
Disallow: /
User-agent: Memo
Disallow: /
User-agent: Metacarta
Disallow: /
User-agent: Mercator
Disallow: /
User-agent: MIDown\ tool
Disallow: /
User-agent: MFC_Tear_Sample
Disallow: /
User-agent: Mirror
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Mister\ PiX
Disallow: /
User-agent: NationalDirectory\ WebSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Nikto
Disallow: /
User-agent: Navroad
Disallow: /
User-agent: NearSite
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NetResearchServer
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: Net\ Vampire
Disallow: /
User-agent: Net\ Probe
Disallow: /
User-agent: NetZip
Disallow: /
User-agent: nexuscache
Disallow: /
User-agent: Ninja
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: our\ agent
Disallow: /
User-agent: onestop
Disallow: /
User-agent: oBot
Disallow: /
User-agent: Octopus
Disallow: /
User-agent: Offline\ Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind\ data\ gatherer
Disallow: /
User-agent: PageGrabber
Disallow: /
User-agent: Papa\ Foto
Disallow: /
User-agent: Xaldon\ WebSpider
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: Xara
Disallow: /
User-agent: Y!TunnelPro
Disallow: /
User-agent: YahooYSMcm
Disallow: /
User-agent: Zade
Disallow: /
User-agent: ZBot
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Rogerbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Dotbot
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: Xaldon\ WebSpider
Disallow: /
User-agent: Screaming Frog SEO Spider/8.1
Disallow: /
User-agent: DoBot
Disallow: /
User-agent: Laserlikebot
Disallow: /
User-agent: oBot
Disallow: /
User-agent: Neliveto
Disallow: /
User-Agent: ahrefs
Disallow: /
User-Agent: AHC/1.0
Disallow: /
User-Agent: AHC
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Disallow: /
User-agent: evc-batch/2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Disallow: /
User-agent: SemrushBot/1.2~bl
Disallow: /
User-agent: evc-batch
Disallow: /
User-agent: 007ac9 Crawler
disallow: /
User-agent: 1 2 3 Submit PRO
disallow: /
User-agent: 200PleaseBot
disallow: /
User-agent: 2ADAMbot
disallow: /
User-agent: 2ADAMbot/1.0
disallow: /
User-agent: 360Spider
disallow: /
user-agent: Abonti
disallow: /
user-agent: Abonti/0.92
disallow: /
user-agent: abot v1.0
disallow: /
user-agent: aboutthedomain
disallow: /
user-agent: Add Catalog
disallow: /
user-agent: Add Catalog/2.1
disallow: /
user-agent: AdvBot
disallow: /
user-agent: AdvBot/2.0
disallow: /
user-agent: AhrefsBot
disallow: /
user-agent: Ahrefs-Bot
disallow: /
user-agent: AhrefsBot/1.0
disallow: /
user-agent: Ahrefs-Bot/1.0
disallow: /
user-agent: Ahrefs-Bot/2.0
disallow: /
user-agent: Ahrefs-Bot/3.0
disallow: /
user-agent: Ahrefs-Bot/4.0
disallow: /
user-agent: Ahrefs-Bot/5.0
disallow: /
user-agent: aiHitBot
disallow: /
user-agent: aiHitBot/2.9
disallow: /
user-agent: Anonymous/0.0
disallow: /
user-agent: Arachnida
disallow: /
user-agent: Associative Spider
disallow: /
User-agent: Baiduspider
disallow: /
User-agent: Baidu Spider
disallow: /
User-agent: Battleztar Bazinga
disallow: /
User-agent: Battleztar Bazinga/0.01
disallow: /
User-agent: BDFetch
disallow: /
User-agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
Disallow: /
User-agent: ZmEu
Disallow: /
User-agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
Disallow: /
User-agent: betaBot
disallow: /
User-agent: bieshu
disallow: /
User-agent: Bigli SEO
disallow: /
User-agent: Blackboard Safeassign
disallow: /
User-agent: Blazer 1.0
disallow: /
User-agent: BLEXBot
disallow: /
User-agent: BLEXBot/1.0
disallow: /
User-agent: BLP_bbot
disallow: /
User-agent: BLP_bbot/0.1
disallow: /
User-agent: BOIA-Accessibility-Agent/PR 1.0
disallow: /
User-agent: BOT for JCE
disallow: /
User-agent: BOT/0.1 (BOT for JCE)
disallow: /
User-agent: BPImageWalker
disallow: /
User-agent: BPImageWalker/2.0
disallow: /
User-agent: BUbiNG
disallow: /
User-agent: BuiBui-Bot
disallow: /
User-agent: BuiBui-Bot/1.0
disallow: /
User-agent: ca-crawler
disallow: /
User-agent: ca-crawler/1.0
disallow: /
User-agent: CakePHP
disallow: /
User-agent: Calypso v/0.01
disallow: /
User-agent: Calypso
disallow: /
User-agent: CB/Nutch-1.7
disallow: /
User-agent: CCBot
disallow: /
User-agent: CCBot/2.0
disallow: /
User-agent: Checkbot
disallow: /
User-agent: checkgzipcompression.com
disallow: /
User-agent: chushou
disallow: /
User-agent: CloudServerMarketSpider
disallow: /
User-agent: CloudServerMarketSpider/1.0
disallow: /
User-agent: Clushbot/3.x-BinaryFury
disallow: /
User-agent: CMS Crawler
disallow: /
User-agent: CMS Crawler: http://www.cmscrawler.com
disallow: /
User-agent: coccoc
disallow: /
User-agent: CoinCornerBot
disallow: /
User-agent: CoinCornerBot/1.1
disallow: /
User-agent: Copyscape
disallow: /
User-agent: crawler4j
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.0
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.1
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.7
disallow: /
User-agent: CrazyWebCrawler
disallow: /
User-agent: CrazyWebCrawler-Spider
disallow: /
User-agent: Crowsnest
disallow: /
User-agent: Crowsnest/0.5
disallow: /
User-agent: Curious George - www.analyticsseo.com/crawler
disallow: /
User-agent: Curious George
disallow: /
User-agent: cuwhois
disallow: /
User-agent: cuwhois/1.0
disallow: /
User-agent: dahoms
disallow: /
User-agent: datagnionbot
disallow: /
User-agent: DeuSu/5.0.2
disallow: /
User-agent: Digincore
disallow: /
User-agent: Digincore bot
disallow: /
User-agent: Dispatch/0.11.0
disallow: /
User-agent: Domain Re-Animator Bot
disallow: /
User-agent: DomainAppender /1.0
disallow: /
User-agent: DomainAppender
disallow: /
User-agent: DomainCrawler/3.0
disallow: /
User-agent: DomainSigmaCrawler
disallow: /
User-agent: DomainSigmaCrawler/0.1
disallow: /
User-agent: Domnutch
disallow: /
User-agent: Domnutch-Bot
disallow: /
User-agent: Domnutch-Bot/Nutch
disallow: /
User-agent: Domnutch-Bot/Nutch-1.0
disallow: /
User-agent: dotbot
disallow: /
User-agent: ECCP/1.2.1
disallow: /
User-agent: eCommerceBot
disallow: /
User-agent: enlle punto com/Nutch-1.9
disallow: /
User-agent: EPiServer Link Checker
disallow: /
User-agent: EuripBot
disallow: /
User-agent: EuripBot/2.0
disallow: /
User-agent: evc/2.0
disallow: /
User-agent: evc-batch
disallow: /
User-agent: evc-batch/2.0
disallow: /
User-agent: Express WebPictures
disallow: /
User-agent: Faraday v0.8.8
disallow: /
User-agent: Faraday
disallow: /
User-agent: Findxbot
disallow: /
User-agent: Findxbot/1.0
disallow: /
User-agent: Flamingo_SearchEngine
disallow: /
User-agent: Flipboard Robot
disallow: /
User-agent: GetProxi.es-bot
disallow: /
User-agent: GetProxi.es-bot/1.1
disallow: /
User-agent: GigablastOpenSource
disallow: /
User-agent: GigablastOpenSource/1.0
disallow: /
User-agent: Girafabot
disallow: /
User-agent: Gluten Free Crawler
disallow: /
User-agent: Gluten Free Crawler/1.0
disallow: /
User-agent: GriffinBot
disallow: /
User-agent: GrifinBot/0.01
disallow: /
User-agent: GWPImages
disallow: /
User-agent: GWPImages/1.0
disallow: /
User-agent: Haiula
disallow: /
User-agent: Haiula/1.4
disallow: /
User-agent: HaosouSpider
disallow: /
User-agent: Hivemind
disallow: /
User-agent: HostHarvest
disallow: /
User-agent: HostHarvest/0.4.28
disallow: /
User-agent: HRCrawler
disallow: /
User-agent: HRCrawler/2.0
disallow: /
User-agent: http://git.io/tl_S2w
disallow: /
User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
disallow: /
User-agent: HubSpot Links Crawler 1.0
disallow: /
User-agent: HubSpot Webcrawler
disallow: /
User-agent: HubSpot
disallow: /
User-agent: hunchan
disallow: /
User-agent: HyperCrawl
disallow: /
User-agent: HyperCrawl/0.2
disallow: /
User-agent: ICAP-IOD
disallow: /
User-agent: ICC-Crawler
disallow: /
User-agent: ICC-Crawler/2.0
disallow: /
User-agent: Ichiro Robot
disallow: /
User-agent: image.coccoc/1.0
disallow: /
User-agent: Image2play
disallow: /
User-agent: Image2play/0.1
disallow: /
User-agent: Indy Library
disallow: /
User-agent: InsightsCollector
disallow: /
User-agent: InsightsCollector/0.1
disallow: /
User-agent: InsightsCollector/0.1beta
disallow: /
User-agent: integrity/5
disallow: /
User-agent: InterNaetBoten
disallow: /
User-agent: InterNaetBoten/0.99
disallow: /
User-agent: Barkrowler/0.7 (+http://www.exensa.com/crawl)
disallow: /
User-agent: Barkrowler
disallow: /
User-agent: IRL Crawler
disallow: /
User-agent: James BOT - WebCrawler
disallow: /
User-agent: James BOT
disallow: /
User-agent: JamesBOT
disallow: /
User-agent: JetBrains 5.0
disallow: /
User-agent: JetBrains
disallow: /
User-agent: Kraken
disallow: /
User-agent: Kraken/0.1
disallow: /
User-agent: Kyoto-Tohoku-Crawler/v1
disallow: /
User-agent: larbin
disallow: /
User-agent: lechenie
disallow: /
User-agent: libwww-perl
disallow: /
User-agent: link checker
disallow: /
User-agent: Link/1.0
disallow: /
User-agent: linkCheck
disallow: /
User-agent: linkCheckV3.0
disallow: /
User-agent: Linkdex
disallow: /
User-agent: linkdex.com/v2.0
disallow: /
User-agent: linkdex.com/v2.1
disallow: /
User-agent: LinkdexBot
disallow: /
User-agent: linkdexbot/2.0
disallow: /
User-agent: linkdexbot/2.1
disallow: /
User-agent: linkdexbot-mobile/2.1
disallow: /
User-agent: LinkpadBot
disallow: /
User-agent: LinkpadBot/1.06
disallow: /
User-agent: LinqiaScrapeBot
disallow: /
User-agent: LinqiaScrapeBot/1.0
disallow: /
User-agent: Lipperhey SEO Service
disallow: /
User-agent: Lipperhey
disallow: /
User-agent: Lipperhey-Kaus-Australis
disallow: /
User-agent: Lipperhey-Kaus-Australis/5.0
disallow: /
User-agent: listicka
disallow: /
User-agent: LSSRocketCrawler
disallow: /
User-agent: LSSRocketCrawler/1.0 LightspeedSystems
disallow: /
User-agent: LSSRocketCrawler/1.0
disallow: /
User-agent: ltx71
disallow: /
User-agent: LWNutch/Nutch-1.4
disallow: /
User-agent: Mail.RU
disallow: /
User-agent: Mail.RU_Bot
disallow: /
User-agent: Mail.RU_Bot/2.0
disallow: /
User-agent: Mail.RU_Bot/Fast/2.0
disallow: /
User-agent: md5sum
disallow: /
User-agent: md5sum\x22
disallow: /
User-agent: meanpathbot
disallow: /
User-agent: MegaIndex.ru
disallow: /
User-agent: MegaIndex.ru/2.0
disallow: /
User-agent: mezhpozvonochnoi
disallow: /
User-agent: Mike-Crawler
disallow: /
User-agent: MixBot
disallow: /
User-agent: MixrankBot
disallow: /
User-agent: MJ12bot
disallow: /
User-agent: Monkeybot/0.1
disallow: /
User-agent: my crawler
disallow: /
User-agent: My Nutch Spider/Nutch-1.9
disallow: /
User-agent: mycrowl/Nutch-1.9
disallow: /
User-agent: MyGreatUA/2.0
disallow: /
User-agent: MyIPTest
disallow: /
User-agent: NameProtect Robot
disallow: /
User-agent: NerdyBot
disallow: /
User-agent: Netcraft Spider
disallow: /
User-agent: netEstate NE Crawler
disallow: /
User-agent: NetLyzer FastProbe
disallow: /
User-agent: NetResearchServer
disallow: /
User-agent: NetResearchServer/4.0
disallow: /
User-agent: Nmap Scripting Engine
disallow: /
User-agent: node.io
disallow: /
User-agent: node.js
disallow: /
User-agent: Node/simplecrawler 0.5.2
disallow: /
User-agent: Node/simplecrawler
disallow: /
User-agent: oBot/2.3.1
disallow: /
User-agent: omgilibot
disallow: /
User-agent: omgilibot/0.4
disallow: /
User-agent: Online Domain Tools - Online Website Link Checker
disallow: /
User-agent: Online Domain Tools - Online Website Link Checker/1.2
disallow: /
User-agent: Openfind Robot
disallow: /
User-agent: OpenHoseBot
disallow: /
User-agent: OpenHoseBot/2.1
disallow: /
User-agent: Openstat
disallow: /
User-agent: Openstat/0.1
disallow: /
User-agent: OptimizationCrawler
disallow: /
User-agent: OptimizationCrawler/0.2
disallow: /
User-agent: Page Analyzer v4.0
disallow: /
User-agent: Page Analyzer
disallow: /
User-agent: PageAnalyzer
disallow: /
User-agent: PageAnalyzer/1.1
disallow: /
User-agent: PageAnalyzer/1.5
disallow: /
User-agent: PagesInventory
disallow: /
User-agent: Pagespeed/1.1 Fetcher
disallow: /
User-agent: Pagespeed/1.1
disallow: /
User-agent: Pagespeedbot
disallow: /
User-agent: Perl LWP
disallow: /
User-agent: PHPCrawl
disallow: /
User-agent: phpSiteCheck 1.0
disallow: /
User-agent: phpSiteCheck
disallow: /
User-agent: Plukkie
disallow: /
User-agent: POGS/2.0
disallow: /
User-agent: Powermarks
disallow: /
User-agent: PowerPivot
disallow: /
User-agent: PRIVACY_ENFAQ.jsp
disallow: /
User-agent: Prlog
disallow: /
User-agent: Prlog/1.0
disallow: /
User-agent: publiclibraryarchive.org
disallow: /
User-agent: publiclibraryarchive.org/1.0
disallow: /
User-agent: Pu_iN Crawler
disallow: /
User-agent: Putin
disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /
User-agent: Putin spider
disallow: /
User-agent: qingdao
disallow: /
User-agent: QlikView
disallow: /
User-agent: quipu
disallow: /
User-agent: quipu/1.0
disallow: /
User-agent: quipu/2.0
disallow: /
User-agent: R6_CommentReader
disallow: /
User-agent: R6_FeedFetcher
disallow: /
User-agent: Riddler
disallow: /
User-agent: RivalSeek.com-Bot
disallow: /
User-agent: rogerbot
disallow: /
User-agent: rogerbot/1.0
disallow: /
User-agent: rootlink
disallow: /
User-agent: RU_Bot/2.0
disallow: /
User-agent: Scopia
disallow: /
User-agent: Scopia crawler
disallow: /
User-agent: Scopia crawler 1.0
disallow: /
User-agent: Scopia crawler 1.1
disallow: /
User-agent: Scopia crawler 1.2
disallow: /
User-agent: Scrapy
disallow: /
User-agent: Scrapy/0.16.5
disallow: /
User-agent: Scrapy/0.24.4
disallow: /
User-agent: Scrapy/0.24.5
disallow: /
User-agent: Scrapy/0.24.6
disallow: /
User-agent: Scrapy/1.0.1
disallow: /
User-agent: Screaming Frog SEO Spider
disallow: /
User-agent: Screaming Frog SEO Spider/2,55
disallow: /
User-agent: Screaming Frog SEO Spider/2.55
disallow: /
User-agent: Screaming Frog SEO Spider/3.1
disallow: /
User-agent: Screaming Frog SEO Spider/3.3
disallow: /
User-agent: Screaming Frog SEO Spider/4.1
disallow: /
User-agent: Screaming Frog SEO Spider/5.0
disallow: /
User-agent: Screaming Frog SEO Spider/5.1
disallow: /
User-agent: Screaming Frog SEO Spider/5.1 Beta 2
disallow: /
User-agent: scrutiny/4
disallow: /
User-agent: SemrushBot
disallow: /
User-agent: SemrushBot-SA
disallow: /
User-agent: SEOdiver/1.0
disallow: /
User-agent: SEOkicks
disallow: /
User-agent: SEOkicks-Robot
disallow: /
User-agent: SEOlyticsCrawler
disallow: /
User-agent: SEOlyticsCrawler/3.0
disallow: /
User-agent: seoscanners
disallow: /
User-agent: seoscanners.net/1
disallow: /
User-agent: SEOstats 2.1.0
disallow: /
User-agent: Seosys/Nutch-2.3
disallow: /
User-agent: SetCronJob/1.0
disallow: /
User-agent: R6_FeedFetcher_(www.radian6.com/crawler)
disallow: /
User-agent: SeznamBot
disallow: /
User-agent: SheerBoredom.Experimental.Robot
disallow: /
User-agent: SheerBoredom.Experimental.Robot/0.2
disallow: /
User-agent: ShowyouBot
disallow: /
User-agent: Simplecrawler
disallow: /
User-agent: SISTRIX Crawler
disallow: /
User-agent: sistrix
disallow: /
User-agent: SiteBot
disallow: /
User-agent: SiteBot/0.1
disallow: /
User-agent: SiteExplorer
disallow: /
User-agent: SiteExplorer/1.0
disallow: /
User-agent: SiteExplorer/1.0b
disallow: /
User-agent: Siteluxbot
disallow: /
User-agent: Siteluxbot/1.0
disallow: /
User-agent: SkimBot
disallow: /
User-agent: RSSingBot (http://www.rssing.com)
disallow: /
User-agent: SkimBot/1.0
disallow: /
User-agent: sky nutch crawler/Nutch-1.9
disallow: /
User-agent: SMTBot
disallow: /
User-agent: SMTBot/1.0
disallow: /
User-agent: SNK Screenshot Bot
disallow: /
User-agent: SNK Screenshot Bot/0.20
disallow: /
User-agent: Sogou Spider
disallow: /
User-agent: Sogou web spider
disallow: /
User-agent: SpamBayes
disallow: /
User-agent: SpamBayes/1.1a3+
disallow: /
User-agent: spbot
disallow: /
User-agent: spbot/4.4.2
disallow: /
User-agent: spiderbot
disallow: /
User-agent: SpiderLing
disallow: /
User-agent: Spiderbot/Nutch-1.7
disallow: /
User-agent: spray-can
disallow: /
User-agent: spray-can/1.2.1
disallow: /
User-agent: SSG/3.0
disallow: /
User-agent: Statastico
disallow: /
User-agent: Statastico/4.0
disallow: /
User-agent: Steeler
disallow: /
User-agent: Steeler/3.5
disallow: /
User-agent: Stratagems Kumo
disallow: /
User-agent: Stratagems
disallow: /
User-agent: StudioFACA Search
disallow: /
User-agent: StudioFACA
disallow: /
User-agent: sukibot
disallow: /
User-agent: sukibot_heritrix
disallow: /
User-agent: sukibot_heritrix/3.1.1
disallow: /
User-agent: SurveyBot
disallow: /
User-agent: Synapse
disallow: /
User-agent: Synthesio Crawler release MonaLisa
disallow: /
User-Agent: tbot-nutch/Nutch-1.10
disallow: /
User-Agent: Traackr.com bot
disallow: /
User-Agent: trendictionbot
disallow: /
User-Agent: Trendiction-Bot
disallow: /
User-Agent: TrueBot
disallow: /
User-Agent: TrueBot/1.0
disallow: /
User-Agent: TulipChain/5.xx
disallow: /
User-Agent: TWMBot/0.1
disallow: /
User-Agent: Typhoeus
disallow: /
User-Agent: UCMore Crawler App
disallow: /
User-Agent: uMBot-LN
disallow: /
User-Agent: uMBot-LN/1.0
disallow: /
User-Agent: updown_tester
disallow: /
User-Agent: URLChecker
disallow: /
User-agent: V1.0/1.2
disallow: /
User-agent: w3af.org
disallow: /
User-agent: WASALive
disallow: /
User-agent: WASALive-Bot
disallow: /
User-agent: vBSEO
disallow: /
User-agent: WBSearchBot
disallow: /
User-agent: WBSearchBot/1.1
disallow: /
User-agent: WeAreNotEvil
disallow: /
User-agent: WebAlta
disallow: /
User-agent: WebAlta Crawler
disallow: /
User-agent: Web corpus crawler
disallow: /
User-agent: WebCookies
disallow: /
User-agent: WebCookies/1.0
disallow: /
User-agent: WebCopier vx.xa
disallow: /
User-agent: Webnest 0.9
disallow: /
User-agent: WebQL
disallow: /
User-agent: Webscout
disallow: /
User-agent: Webscout/1.0
disallow: /
User-agent: Web-sniffer
disallow: /
User-agent: Web-sniffer/1.1.0
disallow: /
User-agent: Webster Pro V3.4
disallow: /
User-agent: WebTarantula.com Crawler
disallow: /
User-agent: WeCrawlForThePeace
disallow: /
User-agent: VegeBot
disallow: /
User-agent: Vegi bot
disallow: /
User-agent: WeLikeLinks
disallow: /
User-agent: VeriCiteCrawler
disallow: /
User-agent: VeriCiteCrawler/Nutch-1.9
disallow: /
User-agent: WhatWeb
disallow: /
User-agent: WhatWeb/0.4.8-dev
disallow: /
User-agent: Visited by http://tools.geek-tools.org
disallow: /
User-agent: voltron
disallow: /
User-agent: woobot
disallow: /
User-agent: woobot/1.1
disallow: /
User-agent: woobot/2.0
disallow: /
User-agent: Vorboss Web Crawler
disallow: /
User-agent: Vorboss Web Crawler/Nutch-2.3
disallow: /
User-agent: WorldBrewBot
disallow: /
User-agent: WorldBrewBot/2.1
disallow: /
User-agent: worldwebheritage.org
disallow: /
User-agent: worldwebheritage.org/1.0
disallow: /
User-agent: wscheck.com
disallow: /
User-agent: wscheck.com/1.0.0
disallow: /
User-agent: www.deadlinkchecker.com
disallow: /
User-agent: www.petitsage.fr site detector 0.4
disallow: /
User-agent: WWW-Mechanize
disallow: /
User-agent: WWW-Mechanize/1.74
disallow: /
User-agent: Xenu Link Sleuth
disallow: /
User-agent: Xenu's Link Sleuth
disallow: /
User-agent: XoviBot
disallow: /
User-agent: XoviBot/2.0
disallow: /
User-agent: XSpider
disallow: /
User-agent: Yandex Robot
disallow: /
User-agent: Yandex
disallow: /
User-agent: Yetibot
disallow: /
User-agent: YisouSpider
disallow: /
User-agent: yoozBot
disallow: /
User-agent: yoozBot-2.2
disallow: /
User-agent: zgrab/0.x
disallow: /
User-agent: zzabmbot
disallow: /
User-agent: zzabmbot/1.0
disallow: /
User-agent: Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-2.3)
Disallow: /
User-agent: Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Disallow: /
User-agent: Gigabot/2.0
Disallow: /
User-agent: GameSpyHTTP/1.0
Disallow: /
User-agent: Baiduspider ( http://www.baidu.com/search/spider.htm)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Disallow: /
User-Agent: istellabot/t.1.13
Disallow: /
User-Agent: MagpieRSS/0.72 (+http://magpierss.sf.net)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /
User-Agent: Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
Disallow: /
User-Agent: Liferea
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-Agent: ahrefsbot
Disallow: /
User-Agent: Ahrefs
Disallow: /
User-Agent: Ahrefsbot
Disallow: /
User-Agent: BoardReader Blog Indexer(http://boardreader.com)
Disallow: /
User-Agent: BoardReader
Disallow: /
User-Agent: YOUBot
Disallow: /
User-Agent: AhrefsBot.com
Disallow: /
User-Agent: yandexBot
Disallow: /
User-Agent: Dow Jones Searchbot
Disallow: /
User-Agent: TurnitinBot
Disallow: /
User-Agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /
User-Agent: Almaden
Disallow: /
User-Agent: eCairn-Grabber/1.0
Disallow: /
User-Agent: eCairn-Grabber
Disallow: /
User-Agent: Amazon
Disallow: /
User-Agent: Gluten Free Crawler/1.0
Disallow: /
User-Agent: DotBot/1.1
Disallow: /
User-Agent: LinqiaCrawlerBot/1.0
Disallow: /
User-Agent: istellabot/t.1.13
Disallow: /
User-Agent: nutch-1.4/Nutch-1.4
Disallow: /
User-Agent: nutch-1.4
Disallow: /
User-Agent: DoBot
Disallow: /
User-Agent: AsapBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: coccocbot-web
Disallow: /
User-Agent: amazonbot
Disallow: /
User-Agent: Amazon-bot
Disallow:/
User-Agent: Meditoolkitbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Disallow: /
User-agent: Yeti/1.1
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: Yahoo! Slurp
Disallow: /
User-agent: HaoSouSpider
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Applebot
Disallow: /
User-agent: Applebot/0.1; +http://www.apple.com/go/applebot
Disallow: /
User-agent: Applebot/0.1
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Naverbot
Disallow: /
User-agent: seznambot
Disallow: /
User-agent: teoma
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: EtaoSpider
Disallow: /
User-Agent: swebot
Disallow: /
User-Agent: Newsbrain
Disallow: /
User-Agent: Feedreader 3.14
Disallow: /
User-Agent: Feedreader
Disallow: /
User-Agent: Feedreader 3.14 (Powered by Newsbrain)
Disallow: /
User-agent: aihitbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Exabot-Thumbnails
Disallow: /
User-agent: sitebot
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: TwengaBot-Discover
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Bender
Disallow: /
User-agent: discobot
Disallow: /
User-Agent: panscient.com
Disallow: /
User-Agent: searchwebengine.net
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: NerdByNature.Bot
disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: sistrix
Disallow: /
User-agent: SindiceBot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: IWAgent/1.0
Disallow: /
User-agent: IWAgent
Disallow: /
User-agent: BotALot
Disallow:/
User-agent: findfiles.net
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Goodzer
Disallow: /
User-agent: DJEcoBot
Disallow: /
User-agent: DJEcoBot/1.0
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
User-agent: AhrefsBot/5.2
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: spray-can/1.3.4
Disallow: /
User-agent: spray-can
Disallow: /
User-agent: spray can
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: 360spider
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: BacklinkCrawler
Disallow: /
User-agent: lemurwebcrawler
Disallow: /
User-agent: gigabot
Disallow: /
User-agent: discobot Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: FAST Enterprise Crawler 6
Disallow: /
User-agent: Sensis.com.au Web Crawler
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: XilopixBot/0.1
Disallow: /
User-agent: XilopixBot
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Arachmo
Disallow: /
User-agent: Acoon
Disallow: /
User-agent: adbeat_bot
Disallow: /
User-agent: AgentLinkSpammer
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: worio bot heritrix
Disallow: /
User-agent: trovitBot
Disallow: /
User-agent: MojeekBot/0.6
Disallow: /
User-agent: rogerbot/1.0
Disallow: /
User-agent: istellabot/t.1.13
Disallow: /
User-agent: SeznamBot/3.2
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: careerbot
Disallow: /
User-agent: SogouBot
Disallow: /
User-agent: Sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogouspider
Disallow: /
User-agent: Qihoo360Bot
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: COMODOSpider
Disallow: /
User-agent: findlinks
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: KaloogaBot
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: Mail.RU
Disallow: /
User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Disallow: /
User-agent: SurdotlyBot/1.0
Disallow: /
User-agent: SurdotlyBot
Disallow: /
User-agent: Apexoo Spider (http://www.apexoo.com/spider/)
Disallow: /
User-agent: Apexoo Spider 1.0
Disallow: /
User-agent: Apexoo Spider 1.1
Disallow: /
User-agent: AportCatalogRobot/2.0
Disallow: /
User-agent: Apostrophe/0.2.1.93 (compatible) filecache
Disallow: /
User-agent: Apostrophe
Disallow: /
User-agent: Aplix HTTP/1.0.1
Disallow: /
User-agent: moget
Disallow: /
User-agent: Ichiro
Disallow: /
User-agent: Ichiro 3.0
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: NetcraftSurveyAgent/1.0
Disallow: /
User-agent: OpenWebIndex/Nutch-1.6
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: panoptaStudyBot
Disallow: /
User-agent: checks.panopta.com
Disallow: /
User-agent: psbot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: SeznamBot/1.0
Disallow: /
User-agent: SeznamBot/1.1
Disallow: /
#SeznamBot/3.0
User-agent: SeznamBot/3.0
Disallow: /
User-agent: SistrixCrawler
Disallow: /
User-agent: Sistrix
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SISTRIX
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sosospider+
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sosospider/2.0
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: Wada.vn
Disallow: /
User-agent: Wada.vn Vietnamese Search
Disallow: /
User-agent: Wada.vn Vietnamese Search/2.1
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: YandexBot/3.0-MirrorDetector
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: YandexSomething
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: YoudaoBot/1.0
Disallow: /
User-agent: Zao
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs
Disallow: /
User-agent: aipbot
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: asterias
Disallow:/
User-agent: Aqua_Products
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: becomebot
Disallow: /
User-agent: Black Hole
Disallow:/
User-agent: BlowFish
Disallow:/
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite
Disallow:/
User-agent: CherryPickerSE
Disallow:/
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: FairAd Client
Disallow: /
User-agent: Fasterfox
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow:/
User-agent: Gaisbot
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: Harvest
Disallow:/
User-agent: Go!Zilla
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: HTTrack
Disallow: /
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: Iron33
Disallow: /
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density
Disallow:/
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: Mata Hari
Disallow: /
User-agent: MIIxpc
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NetMechanic
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: Oracle Ultra Search
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: ProPowerBot
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: psbot
Disallow: /
User-agent: QueryN Metasearch
Disallow:/
User-agent: Radiation Retriever
Disallow: /
User-agent: RepoMonkey
Disallow:/
User-agent: Bait
Disallow:/
User-agent: RMA
Disallow:/
User-agent: searchpreview
Disallow: /
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz
Disallow:/
User-agent: SurveyBot
Disallow: /
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: turingos
Disallow:/
User-agent: TurnitinBot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URLy Warning
Disallow:/
User-agent: URL_Spider_Pro
Disallow: /
User-agent: VCI
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebCapture
Disallow: /
User-agent: scalaj-http/1.0
Disallow: /
User-agent: WebCollage
Disallow: /
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: Wget
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
User-agent: Meltawer
Disallow: /
User-agent: Digimind
Disallow: /
User-agent: Knowings
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: Cision
Disallow: /
User-agent: Talkwater
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: kbcrawl
Disallow: /
User-agent: AmiSoftware
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: Ask n read
Disallow: /
User-agent: Qwam content intelligence
Disallow: /
User-agent: Zite
Disallow: /
User-agent: flipboard
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Youmag
Disallow: /
User-agent: Synthesio
Disallow: /
User-agent: trendybuzz
Disallow: /
User-agent: spotter
Disallow: /
User-agent: scoop.it
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: 5emeRue
Disallow: /
User-agent: Augure
Disallow: /
User-agent: Corporama
Disallow: /
User-agent: readability.com
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver-web.archive.org
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: libwww
Disallow: /
User-agent: wget
Disallow: /
User-agent: 5erue
Disallow: /
User-agent: Sysomos/1.0
Disallow: /
User-agent: adequat
Disallow: /
User-agent: adequat-systems
Disallow: /
User-agent: auramundi
Disallow: /
User-agent: coexel
Disallow: /
User-agent: ellisphere
Disallow: /
User-agent: leadbox
Disallow: /
User-agent: mention
Disallow: /
User-agent: Moreover
Disallow: /
User-agent: mytwip
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: mixnode.com
Disallow: /
User-agent: opinion-tracker
Disallow: /
User-agent: proxem
Disallow: /
User-agent: score3
Disallow: /
User-agent: trendeo
Disallow: /
User-agent: vecteurplus
Disallow: /
User-agent: verticalsearch
Disallow: /
User-agent: vsw
Disallow: /
User-agent: winello
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: infoseek
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: asknread.com
Disallow: /
User-agent: ellisphere
Disallow: /
User-agent: spotter
Disallow: /
User-agent: LexxeBot/1.0
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; spbot/2.0; http://www.seoprofiler.com/bot/ )
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Nuzzel
Disallow: /
User-agent: Nuzzel Bot
Disallow: /
User-agent: Sosospider+(+http://help.soso.com/webspider.htm)
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: SiteBot/0.1
Disallow: /
User-agent: SiteBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: crystalsemanticsbot
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: NetSeer crawler
Disallow: /
User-agent: trovitBot
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: discobot
Disallow: /
User-agent: Jyxobot
Disallow: /
User-agent: sogou
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogoubot
Disallow: /
User-agent: Sogou.com
Disallow: /
User-agent: sogouspider-106-120-173-138.crawl.sogou.com
Disallow: /
User-agent: crawl13.exabot.com
Disallow: /
User-agent: sogouspider-106-120-173-138
Disallow: /
User-agent: Sogou web spider/4.0
Disallow: /
User-agent: Sogouwebspider
Disallow: /
User-agent: sogouspider
Disallow: /
User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Disallow: /
User-agent: Exabot/3.0
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: sistrix
Disallow: /
User-agent: heritrix
Disallow: /
User-agent: GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: NerdByNature.Bot
Disallow: /
User-agent: psbot
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: AddThis.com robot tech.support@clearspring.com
Disallow: /
User-agent: AddThis.com
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: proximic
Disallow: /
User-agent: discoverybot
Disallow: /
User-agent: bl.uk_lddc_bot
Disallow: /
User-agent: IstellaBot
Disallow: /
User-agent: seokicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: Bender
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Yasni
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: netEstate NE Crawler
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Pixray-Seeker
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: integromedb
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: BDCbot
Disallow: /
User-agent: GrapeshotCrawler
Disallow: /
User-agent: WeSEE:Search
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: admantx
Disallow: /
User-agent: spbot
Disallow: /
User-agent: BUbiNG
Disallow: /
User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: Harvest/1.5
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Nutch
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: Spiderbot
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: Xenu Link Sleuth/1.3.8
User-agent: yacy
User-agent: yandex
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Spiderbot/Nutch-1.7
User-agent: g2reader-bot/1.0
User-agent: SocialShare/1.0
User-agent: Owlin Bot
User-agent: trendictionbot
Disallow: /
# Disallow: Sistrix
User-agent: sistrix
Disallow: /
# Disallow: Sistrix
User-agent: SISTRIX Crawler
Disallow: /
# Disallow: Sistrix
User-agent: SISTRIX
Disallow: /
# Disallow: SEOkicks-Robot
User-agent: SEOkicks-Robot
Disallow: /
# Disallow: jobs.de-Robot
User-agent: jobs.de-Robot
Disallow: /
# Backlink Analysis
user-agent: AhrefsBot
disallow: /
# Bot der Leipziger Unister Holding GmbH
user-agent: UnisterBot
disallow: /
# http://moz.com/products
User-agent: dotbot
Disallow: /
# http://www.searchmetrics.com
User-agent: SearchmetricsBot
Disallow: /
# http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
User-agent: MJ12bot
Disallow: /
# http://www.domaintools.com/webmasters/surveybot.php
User-agent: SurveyBot
Disallow: /
# http://www.seodiver.com/bot
user-agent: SEOdiver
disallow: /
# http://openlinkprofiler.org/bot
User-agent: spbot
Disallow: /
# http://www.wotbox.com/bot/
User-agent: wotbox
Disallow: /
# http://www.opensiteexplorer.org/dotbot
# http://moz.com/researchtools/ose/dotbot
User-agent: dotbot
Disallow: /
# http://www.meanpath.com/meanpathbot.html
User-agent: meanpathbot
Disallow: /
# http://www.backlinktest.com/crawler.html
User-agent: BacklinkCrawler
Disallow: /
# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler
Disallow: /
# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Disallow: /
# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1
Disallow: /
# http://filterdb.iss.net/crawler/
User-agent: oBot
Disallow: /
User-agent: fr-crawler
Disallow: /
# http://webmeup-crawler.com
User-agent: BLEXBot
Disallow: /
# https://megaindex.com/crawler
User-agent: MegaIndex.ru
Disallow: /
User-agent: megaindex.com
Disallow: /
# http://www.cloudservermarket.com
User-Agent: CloudServerMarketSpider
Disallow: /
# http://www.trendiction.de/de/publisher/bot
User-Agent: trendictionbot
Disallow: /
# http://www.exalead.com
User-agent: Exabot
Disallow: /
# http://www.career-x.de/bot.html
User-agent: careerbot
Disallow: /
# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /
User-agent: seoscanners.net/1
Disallow: /
User-agent: MetaJobBot
Disallow: /
User-agent: Spiderbot
Disallow: /
User-agent: LinkStats
Disallow: /
User-agent: JobboerseBot
Disallow: /
User-agent: ICCrawler
Disallow: /
User-agent: Plista
Disallow: /
User-agent: Domain Re-Animator Bot
Disallow: /
User-agent: Aboundexbot
Disallow: /
User-agent: acapbot
Disallow: /
User-agent: AdnormCrawler
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: AOLbot/4.0
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: Attribot
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: Bot.AraTurka.com
Disallow: /
User-agent: Butterfly/1.0
Disallow: /
User-agent: BuzzSumo
Disallow: /
User-agent: cis455crawler
Disallow: /
User-agent: ClarityBot
Disallow: /
User-agent: ClarityDailyBot
Disallow: /
User-agent: CMS Crawler
Disallow: /
User-agent: CommonCrawler
Disallow: /
User-agent: crawler4j (http://code.google.com/p/crawler4j/)
Disallow: /
User-agent: CrazyWebCrawler
Disallow: /
User-agent: CrazyWebCrawler-Spider
Disallow: /
User-agent: DataparkSearch
Disallow: /
User-agent: deepcrawl
Disallow: /
User-agent: demandbase-bot
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: dialogsearch.com
Disallow: /
User-agent: diffbot
Disallow: /
User-agent: dj-research
Disallow: /
User-agent: DomainAppender
Disallow: /
User-agent: DomainMacroCrawler
Disallow: /
User-agent: DomainSigmaCrawler
Disallow: /
User-agent: Domain Re-Animator Bot
Disallow: /
User-agent: Ezooms Robot
Disallow: /
User-agent: Experibot
Disallow: /
User-agent: FavOrg
Disallow: /
# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /
# https://turnitin.com/robot/crawlerinfo.html
User-agent: turnitinbot
Disallow: /
# http://help.coccoc.com/
User-agent: coccoc
Disallow: /
# ubermetrics-technologies.com
User-agent: um-IC
Disallow: /
User-agent: scanbot
Disallow: /
# datenbutler.de
User-agent: mindUpBot
Disallow: /
# http://searchgears.de/uber-uns/crawling-faq.html
User-agent: sg-Orbiter
Disallow: /
# http://commoncrawl.org/faq/
User-agent: CCBot
Disallow: /
# http://linkfluence.net/
User-agent: Kraken
Disallow: /
# http://www.botje.com/plukkie.htm
User-agent: plukkie
Disallow: /
# https://www.safedns.com/searchbot
User-agent: SafeDNSBot
Disallow: /
# http://www.haosou.com/help/help_3_2.html
User-agent: 360Spider
Disallow: /
# http://www.haosou.com/help/help_3_2.html
User-agent: HaosouSpider
Disallow: /
# http://www.moz.com/dp/rogerbot
User-agent: rogerbot
Disallow: /
# http://www.openhose.org/bot.html
User-agent: OpenHoseBot
Disallow: /
# http://www.screamingfrog.co.uk/seo-spider/
User-agent: Screaming Frog SEO Spider
Disallow: /
# http://thumbsniper.com
User-agent: ThumbSniper
Disallow: /
# http://www.radian6.com/crawler
User-agent: R6_CommentReader
Disallow: /
User-agent: ImplisenseBot
Disallow: /
# http://cliqz.com/company/cliqzbot
User-agent: Cliqzbot
Disallow: /
# https://www.aihitdata.com/about
User-agent: aiHitBot
Disallow: /
# http://www.trendiction.com/en/publisher/bot
User-Agent: trendictionbot
Disallow: /
# Riddler
# Riddler (http://riddler.io/about)
# http://riddler.io/about
User-agent: Riddler
Disallow: /
# SMTBot
# http://www.similartech.com/smtbot
# Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
User-agent: SMTBot
Disallow: /
# ScoutJet
# http://www.scoutjet.com
# Mozilla/5.0+(compatible;+ScoutJet;++http://www.scoutjet.com/)
# Mozilla/5.0+(compatible;+Blekkobot;+ScoutJet;++http://blekko.com/about/blekkobot)
User-agent: ScoutJet
Disallow: /
# SeeBot
# http://www.seegnify.com/bot
# seebot/2.0 (+http://www.seegnify.com/bot)
User-agent: SeeBot
Disallow: /
# Archive.org Bot
# http://archive.org/details/archive.org_bot
# Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot)
# Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot)
User-agent: archive.org_bot
Disallow: /
# AdBeat Bot
# https://www.adbeat.com/operation_policy
# adbeat_bot
User-agent: adbeat_bot
Disallow: /
# MeanPathBot
# http://www.meanpath.com/meanpathbot.html
# Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
User-agent: meanpathbot
Disallow: /
# Mojeek Bot
# https://www.mojeek.com/bot.html
# Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /
# Speedy Spider / EntireWeb Search
# http://www.entireweb.com
# Speedy+Spider+(http://www.entireweb.com)
User-agent: Speedy Spider
Disallow: /
# GenieO
# http://www.genieo.com/webfilter.html
# Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
User-agent: genieo
Disallow: /
# scrapy.org
# by default ROBOTSTXT_OBEY is set to false
# http://doc.scrapy.org/en/latest/topics/settings.html
User-agent: Scrapy
Disallow: /
# Gigablast Open Source Search Engine/Crawler/Spider
# You can ignore robots.txt
# http://gigablast.com/admin.html
# GigablastOpenSource/1.0
User-agent: GigablastOpenSource
Disallow: /
# Semrush Bot
# http://www.semrush.com/bot.html
# Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html)
User-agent: SemrushBot
Disallow: /
# Proximic
# http://www.proximic.com/info/spider.php
# Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
User-agent: proximic
Disallow: /
# AppleBot
# Used by Siri/Spotlight suggestions (commented out for now)
# http://www.apple.com/go/applebot
# Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot)
# Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot)
# Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot)
User-agent: applebot
Disallow: /
# CCBot
# http://commoncrawl.org/faq/
# CCBot/2.0+(http://commoncrawl.org/faq/)
User-agent: CCBot
Disallow: /
# Plukkie
# http://www.botje.com/plukkie.htm
# Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
User-agent: Plukkie
Disallow: /
# Lipperhey Kaus Australis
# https://www.lipperhey.com/en/about/
# Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
User-agent: lipperhey
Disallow: /
# oBot
# http://filterdb.iss.net/crawler/
# Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
# Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
User-agent: oBot
Disallow: /
# Screaming Frog SEO Spider
# Downloaded software, robots.txt can be ignored
# http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
# Screaming+Frog+SEO+Spider/2.55
# Screaming+Frog+SEO+Spider/3.3
User-agent: Screaming Frog SEO Spider
Disallow: /
# VegeBot
# http://www.exactbot.com/vegebot/index.html
User-agent: VegeBot
Disallow: /
# ScreenerBot
# ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
User-agent: ScreenerBot
Disallow: /
# Garlik from Experian
# GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
User-agent: Garlik
Disallow: /
# BusinessBot: Nathan@lead-caddy.com
User-agent: BusinessBot
Disallow: /
# FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
User-agent: FAST Enterprise
Disallow: /
# Nutch
# http://nutch.apache.org/bot.html
# Nutch+Spider/Nutch-1.5
# mycrowl/Nutch-1.9
# nutch+crawler/Nutch-1.9
# sky+nutch+crawler/Nutch-1.9
# Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
User-agent: Nutch
Disallow: /
# Open Link Profiler
# Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
# http://OpenLinkProfiler.org/bot
User-agent: spbot
Disallow: /
# James Bot
# Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
# http://cognitiveseo.com/bot.html
User-agent: JamesBOT
Disallow: /
# Kraken
# Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;+bot@linkfluence.net)
# http://linkfluence.net/
User-agent: Kraken
Disallow: /
# Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
# http://www.linkdex.com/bots/
User-agent: linkdex
Disallow: /
# Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
# Siteimprove.com
User-agent: SiteCheck-sitecrawl by Siteimprove.com
Disallow: /
# Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
# http://megaindex.com/crawler
User-agent: MegaIndex
Disallow: /
# Prlog
# Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
# http://prlog.ru/
User-agent: Prlog
Disallow: /
# Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
# https://www.aihitdata.com/about
User-agent: AiHitBot
Disallow: /
# KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
# http://www.kingland.com
User-agent: KinglandSystemsCorp
Disallow: /
# Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;+SitemapBot@stmp.com)
# http://www.sitemapbot.com
User-agent: SitemapBot
Disallow: /
# Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
# https://www.deepcrawl.com/bot
User-agent: deepcrawl
Disallow: /
# Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21
# http://www.alexa.com/toolbar
#User-agent: AlexaToolbar
#Disallow: /
# Mozilla/5.0+(compatible;+heritrix/3.3.0-SNAPSHOT-20150302-2206++http://127.0.0.1)
# https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
User-agent: heritrix
Disallow: /
# Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
# http://help.naver.com/robots/
User-agent: Yeti
Disallow: /
# Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,+contact+abuse@inpwrd.com+for+information)
User-agent: NetShelter
Disallow: /
# VSE/1.0+(rabraham@multiview.com)
# LSSRocketCrawler/1.0 LightspeedSystems
# http://lightspeedsystems.com
User-agent: LSSRocketCrawler
Disallow: /
# Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
# http://www.idmarch.org/bot.html
User-agent: idmarch
Disallow: /
# Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0
# Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0+(Following+Mediapartners-Google)
User-agent: Daumoa
Disallow: /
# Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
# https://www.companiesintheuk.co.uk/bot.html
User-agent: CukBot
Disallow: /
# Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
# http://www.expertsearch.nl/spider
User-agent: ExpertSearchSpider
Disallow: /
# Mozilla/5.0+(compatible;+Goodzer/2.0;+crawler@goodzer.com)
User-agent: Goodzer
Disallow: /
# WeSEE
# http://www.wesee.com/
User-agent: WeSEE
Disallow: /
# http://ltx71.com/
# ltx71 - (http://ltx71.com/)
User-agent: ltx71
Disallow: /
# rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,+rogerbot-crawler+shiny@moz.com)
# http://moz.com/help/pro/what-is-rogerbot-
User-agent: rogerbot
Disallow: /
# bitlybot
# http://bit.ly ??
User-agent: bitlybot
Disallow: /
# Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
# http://www.trendiction.de/bot
User-agent: trendictionbot
Disallow: /
# SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
User-agent: SearchMassive
Disallow: /
# spiderbot
User-agent: spiderbot
Disallow: /
# yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
# yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
User-agent: yacybot
Disallow: /
# AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;+help@aboutus.org)
# http://www.aboutus.org/Aboutus:Bot
User-agent: AboutUsBot
Disallow: /
# Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
# http://www.opensiteexplorer.org/dotbot
User-agent: DotBot
Disallow: /
# Mozilla/5.0+(compatible;+BLEXBot/1.0;++http://webmeup-crawler.com/)
# http://webmeup-crawler.com/
User-agent: BLEXBot
Disallow: /
# Xenu Link Sleuth 1.3.4
# Xenu Link Sleuth/1.3.8
User-agent: Xenu Link Sleuth
Disallow: /
# Go language
# Go+1.1+package+http
User-agent: Go 1.1
Disallow: /
# wotbox.com
# http://www.wotbox.com/bot/
# Wotbox/2.01+(+http://www.wotbox.com/bot/)
User-agent: Wotbox
Disallow: /
# yandex
# http://yandex.com/bots
# Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
User-agent: YandexBot
Disallow: /
# Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
# http://help.yahoo.com/help/us/ysearch/slurp
# User-agent: Yahoo! Slurp
# Disallow: /
# Yahoo Japan
# http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
# Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
User-agent: Y!J-ASR
Disallow: /
# domainreanimator.com
User-agent: Domain Re-Animator Bot
Disallow: /
# crazywebcrawler.com
User-agent: CrazyWebCrawler-Spider
Disallow: /
# crazywebcrawler.com
User-agent: CrazyWebCrawler
Disallow: /
# Dow Jones Searchbot
# Mozilla/5.0+(compatible;+Dow+Jones+Searchbot)
User-agent: Dow Jones SearchBot
Disallow: /
# OpenVAS.org
#User-agent: OpenVAS
#Disallow: /
# WeCrawlForThePeace - We are not Evil
User-agent: WeCrawForThePeace
Disallow: /
# WeLikeLinks - WeAreNotEvil
User-agent: WeLikeLinks
Disallow: /
# icevikatam/1.0
#User-agent: icevikatam
#Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Disallow: /
User-agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Disallow: /
User-agent: SocialRankIOBot; http://socialrank.io/about
Disallow: /
User-agent: SocialRankIOBot
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: Curious George
Disallow: /
User-agent: CybEye.com
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: ExB Language Crawler
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Genio
Disallow: /
User-agent: LWNutch
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: RediffNewsBot
Disallow: /
User-agent: SEOENGWorldBot
Disallow: /
User-agent: Scanmine
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShowyouBot
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: WocBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: daumoa
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: libcrawl
Disallow: /
User-agent: linkdex
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: repparser
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: sindice-site-manager
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: sogou
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: yolinkBot
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: wget
Disallow: /
# Block Sogou
User-agent: sogou spider
Disallow: /
# Block SEOkicks
User-agent: SEOkicks-Robot
Disallow: /
# Block BlexBot
User-agent: BLEXBot
Disallow: /
# Block SISTRIX
User-agent: SISTRIX Crawler
Disallow: /
# Block Uptime robot
User-agent: UptimeRobot/2.0
Disallow: /
User-agent: 008
Disallow: /
# Block Ezooms Robot
User-agent: Ezooms Robot
Disallow: /
# Block Perl LWP
User-agent: Perl LWP
Disallow: /
# Block BlexBot
User-agent: BLEXBot
Disallow: /
# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /
# Block WiseGuys Robot
User-agent: WiseGuys Robot
Disallow: /
# Block Turnitin Robot
User-agent: Turnitin Robot
Disallow: /
User-agent: HuaweiSymantecSpider
Disallow: /
User-agent: Purebot
Disallow: /
# Block Heritrix
User-agent: Heritrix
Disallow: /
User-agent: eContext/1.0 (eContext Classification Engine)
Disallow: /
User-agent: eContext/1.0
Disallow: /
User-agent: eContext
Disallow: /
# Block pricepi
User-agent: pimonster
Disallow: /
User-agent: Pimonster
Disallow: /
# Block Searchmetrics Bot
User-agent: SearchmetricsBot
Disallow: /
# Block Eniro
User-agent: ECCP/1.0 (search@eniro.com)
Disallow: /
# Block YandexBot
User-agent: Yandex
Disallow: /
# Block Baidu
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /
# Block SoGou
User-agent: Sogou Spider
Disallow: /
# Block Youdao
User-agent: YoudaoBot
Disallow: /
# Block MegaIndex.ru
User-agent: MegaIndex.ru/2.0
Disallow: /
User-agent: Yandex
Disallow: /
user-agent: AhrefsBot
disallow: /
User-Agent: trendictionbot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: search:robot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: xovi
Disallow: /
User-agent: Cutbot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: LivelapBot
Disallow: /
User-agent: LS Session
Disallow: /
User-agent: MagicBot
Disallow: /
User-agent: AliveAdvisorCrawler/1.7.1
Disallow: /
User-agent: AliveAdvisorCrawler/1.7.1 (whole web crawler; http://search.aliveadvisor.com; crawler@aliveadvisor.com)
Disallow: /
User-agent: madaali.de
Disallow: /
User-agent: memoryBot
Disallow: /
User-agent: whole web crawler
Disallow: /
User-agent: MegaIndex.ru/2.0
Disallow: /
User-agent: MegaIndex.ru/
Disallow: /
User-agent: SputnikBot
Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: wget
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: berlin-fu-cow
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: semager
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: InAGist URL Resolver
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Insitesbot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: IndeedBot
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: MegaIndex.ru/2.0
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: megaIndex.ru
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: ltx71
Disallow: /
User-agent: SputnikBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: seoscanners.net
Disallow: /
User-agent: DeuSu
Disallow: /
User-agent: meds-online24.com
Disallow: /
User-agent: seoscanners
Disallow: /
User-agent: GuzzleHttp
Disallow: /
User-agent: IDBot
Disallow: /
User-agent: PRTGCloudBot
Disallow: /
User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebLyzard Crawler
Disallow: /
User-agent: WebCopier v4.3
Disallow: /
User-agent: Vagabondo
Disallow: /
User-agent: Shim-Crawler
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: NimbleCrawler
Disallow: /
User-agent: hl_ftien_spider_v1.1
Disallow: /
User-agent: EMPAS_ROBOT
Disallow: /
User-agent: EDI/1.4.1 (Edacious & Intelligent Web Crawler)
Disallow: /
User-agent: (compatible; hetrix/1.8.0+http://www.accelobot.com)
Disallow: /
User-agent: WebReaper [info@webreaper.net]
Disallow: /
User-agent: WebReaper [webcreaper@otway.com]
Disallow: /
User-agent: Mozilla/5.0 (compatible; Vagabondo/Kliksafe; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Disallow: /
User-agent: Vagabondo/4.0 (webagent at wise-guys dot nl)
Disallow: /
User-agent: Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Disallow: /
# http://www.wise-guys.nl/
User-agent: webcrawler
Disallow: /
User-agent: ilse
Disallow: /
User-agent: Vagabondo
Disallow: /
User-agent: Bilbo
Disallow: /
User-agent: PiplBot
Disallow: /
User-agent: lwp-trivial/5.810
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: discoverybot
Disallow: /
User-agent: SimplePie
Disallow: /
User-agent: SimplePie/1.1.1
Disallow: /
User-agent: SimplePie/1.0.1
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Ocelli/1.4
Disallow: /
User-agent: Ocelli
Disallow: /
User-agent: ezooms.bot@gmail.com
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Ezooms/1.0
Disallow: /
User-agent: Ezooms/1.0; ezooms.bot@gmail.com
Disallow: /
#ad Crawler: http://www.a6corp.com/a6-web-scraping-policy/
User-agent: A6-Indexer/1.0
Disallow: /
#ad Crawler: http://www.grapeshot.co.uk/crawler.php
User-agent: grapeshot
Disallow: /
# http://www.proximic.com/info/spider.php
User-agent: proximic
Disallow: /
# http://www.jobdigger.nl/spider/
User-agent: JobdiggerSpider
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: rogerbot
Disallow:/
User-agent: SEOkicks-Robot
Disallow:/
User-agent: SMTBot
Disallow:/
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: EnaBot/1.2 (http://www.enaball.com/crawler.html)
Disallow: /
User-agent: EnaBot
Disallow: /
User-agent: Go-http-client/1.1
Disallow: /
User-agent: SMTBot/1.0
Disallow: /
User-agent: e-SocietyRobot
Disallow: /
User-agent: Synthesio
Disallow: /
User-agent: Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Disallow: /
User-agent: FreeFind
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Caliperbot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: SimplePie
Disallow: /
User-agent: SimplePie/1.1.3 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20081219
Disallow: /
User-Agent: trendictionbot
Disallow: /
# http://sentibot.eu/
User-agent: SentiBot
Disallow: /
User-agent: SentiBot
Disallow: /
User-agent: omgili/0.5 +http://omgili.com
Disallow: /
User-agent: Mediatoolkitbot (complaints@mediatoolkit.com)
Disallow: /
User-agent: Mediatoolkitbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /
User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: GrapeshotCrawler
Disallow:/
User-agent: Grapeshot Crawler
Disallow:/
User-agent: Grapeshot
Disallow:/
User-agent: omgili/0.5 +http://omgili.com
Disallow: /
User-agent: omgili
Disallow: /
User-agent: musobot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SimplePie
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: Mozilla/5.0 ResearchBot/0.6.2 (project: flattr-id-meta-tokens) This/1
Disallow: /
User-agent: ResearchBot/0.6.2
Disallow: /
User-agent: ResearchBot
Disallow: /
User-agent: jobs.de-Robot
Disallow: /
User-agent: FreeFind
Disallow: /
User-agent: xingbot
Disallow: /
User-agent: xing bot
Disallow: /
User-agent: xing
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: ChinasoSpider
Disallow: /
User-agent: yisouspider
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Caliperbot
Disallow: /
User-agent: spbot
Disallow: /
user-agent: EveryoneSocialBot
Disallow: /
user-agent: something
Disallow: /
user-agent: ABCdatos BotLink
user-agent: Acme.Spider
user-agent: Ahoy! The Homepage Finder
user-agent: Alkaline
user-agent: Anthill
user-agent: Walhello appie
user-agent: Arachnophilia
user-agent: Arale
user-agent: Araneo
user-agent: AraybOt
user-agent: ArchitextSpider
user-agent: Aretha
user-agent: ARIADNE
user-agent: arks
user-agent: AskJeeves
user-agent: ASpider (Associative Spider)
user-agent: ATN Worldwide
user-agent: Atomz.com Search Robot
user-agent: AURESYS
user-agent: BackRub
user-agent: Bay Spider
user-agent: Big Brother
user-agent: Bjaaland
user-agent: BlackWidow
user-agent: Die Blinde Kuh
user-agent: Bloodhound
user-agent: Borg-Bot
user-agent: BoxSeaBot
user-agent: bright.net caching robot
user-agent: BSpider
user-agent: CACTVS Chemistry Spider
user-agent: Calif
user-agent: Cassandra
user-agent: Digimarc Marcspider/CGI
user-agent: Checkbot
user-agent: ChristCrawler.com
user-agent: churl
user-agent: cIeNcIaFiCcIoN.nEt
user-agent: CMC/0.01
user-agent: Collective
user-agent: Combine System
user-agent: Conceptbot
user-agent: ConfuzzledBot
user-agent: CoolBot
user-agent: Web Core / Roots
user-agent: XYLEME Robot
user-agent: Internet Cruiser Robot
user-agent: Cusco
user-agent: CyberSpyder Link Test
user-agent: CydralSpider
user-agent: Desert Realm Spider
user-agent: DeWeb(c) Katalog/Index
user-agent: DienstSpider
user-agent: Digger
user-agent: Digital Integrity Robot
user-agent: Direct Hit Grabber
user-agent: DNAbot
user-agent: DownLoad Express
user-agent: DragonBot
user-agent: DWCP (Dridus' Web Cataloging Project)
user-agent: e-collector
user-agent: EbiNess
user-agent: EIT Link Verifier Robot
user-agent: ELFINBOT
user-agent: Emacs-w3 Search Engine
user-agent: ananzi
user-agent: esculapio
user-agent: Esther
user-agent: Evliya Celebi
user-agent: FastCrawler
user-agent: Fluid Dynamics Search Engine robot
user-agent: Felix IDE
user-agent: Wild Ferret Web Hopper #1, #2, #3
user-agent: FetchRover
user-agent: fido
user-agent: Hämähäkki
user-agent: KIT-Fireball
user-agent: Fish search
user-agent: Fouineur
user-agent: Robot Francoroute
user-agent: Freecrawl
user-agent: FunnelWeb
user-agent: gammaSpider, FocusedCrawler
user-agent: gazz
user-agent: GCreep
user-agent: GetBot
user-agent: GetURL
user-agent: Golem
user-agent: Grapnel/0.01 Experiment
user-agent: Griffon
user-agent: Gromit
user-agent: Northern Light Gulliver
user-agent: Gulper Bot
user-agent: HamBot
user-agent: Harvest
user-agent: havIndex
user-agent: HI (HTML Index) Search
user-agent: Hometown Spider Pro
user-agent: ht://Dig
user-agent: HTMLgobble
user-agent: Hyper-Decontextualizer
user-agent: iajaBot
user-agent: IBM_Planetwide
user-agent: Popular Iconoclast
user-agent: Ingrid
user-agent: Imagelock
user-agent: IncyWincy
user-agent: Informant
user-agent: InfoSeek Robot 1.0
user-agent: Infoseek Sidewinder
user-agent: InfoSpiders
user-agent: Inspector Web
user-agent: IntelliAgent
user-agent: I, Robot
user-agent: Iron33
user-agent: Israeli-search
user-agent: JavaBee
user-agent: JBot Java Web Robot
user-agent: JCrawler
user-agent: Jeeves
user-agent: JoBo Java Web Robot
user-agent: Jobot
user-agent: JoeBot
user-agent: The Jubii Indexing Robot
user-agent: JumpStation
user-agent: image.kapsi.net
user-agent: Katipo
user-agent: KDD-Explorer
user-agent: Kilroy
user-agent: KO_Yappo_Robot
user-agent: LabelGrabber
user-agent: larbin
user-agent: legs
user-agent: Link Validator
user-agent: LinkScan
user-agent: LinkWalker
user-agent: Lockon
user-agent: logo.gif Crawler
user-agent: Lycos
user-agent: Mac WWWWorm
user-agent: Magpie
user-agent: marvin/infoseek
user-agent: Mattie
user-agent: MediaFox
user-agent: MerzScope
user-agent: NEC-MeshExplorer
user-agent: MindCrawler
user-agent: mnoGoSearch search engine software
user-agent: moget
user-agent: MOMspider
user-agent: Monster
user-agent: Motor
user-agent: Muncher
user-agent: Muninn
user-agent: Muscat Ferret
user-agent: Mwd.Search
user-agent: Internet Shinchakubin
user-agent: NDSpider
user-agent: Nederland.zoek
user-agent: NetCarta WebMap Engine
user-agent: NetMechanic
user-agent: NetScoop
user-agent: newscan-online
user-agent: NHSE Web Forager
user-agent: Nomad
user-agent: The NorthStar Robot
user-agent: nzexplorer
user-agent: ObjectsSearch
user-agent: Occam
user-agent: HKU WWW Octopus
user-agent: OntoSpider
user-agent: Openfind data gatherer
user-agent: Orb Search
user-agent: Pack Rat
user-agent: PageBoy
user-agent: ParaSite
user-agent: Patric
user-agent: pegasus
user-agent: The Peregrinator
user-agent: PerlCrawler 1.0
user-agent: Phantom
user-agent: PhpDig
user-agent: PiltdownMan
user-agent: Pimptrain.com's robot
user-agent: Pioneer
user-agent: html_analyzer
user-agent: Portal Juice Spider
user-agent: PGP Key Agent
user-agent: PlumtreeWebAccessor
user-agent: Poppi
User-agent: iskanie
user-agent: PortalB Spider
user-agent: psbot
user-agent: GetterroboPlus Puu
user-agent: The Python Robot
user-agent: Raven Search
user-agent: RBSE Spider
user-agent: Resume Robot
user-agent: RoadHouse Crawling System
user-agent: RixBot
user-agent: Road Runner: The ImageScape Robot
user-agent: Robbie the Robot
user-agent: ComputingSite Robi/1.0
user-agent: RoboCrawl Spider
user-agent: RoboFox
user-agent: Robozilla
user-agent: Roverbot
user-agent: RuLeS
user-agent: SafetyNet Robot
user-agent: Scooter
user-agent: Sleek
user-agent: Search.Aus-AU.COM
user-agent: SearchProcess
user-agent: Senrigan
user-agent: SG-Scout
user-agent: ShagSeeker
user-agent: Shai'Hulud
user-agent: Sift
user-agent: Simmany Robot Ver1.0
user-agent: Site Valet
user-agent: Open Text Index Robot
user-agent: SiteTech-Rover
user-agent: Skymob.com
user-agent: SLCrawler
user-agent: Inktomi Slurp
user-agent: Smart Spider
user-agent: Snooper
user-agent: Solbot
user-agent: Spanner
user-agent: Speedy Spider
user-agent: spider_monkey
user-agent: SpiderBot
user-agent: Spiderline Crawler
user-agent: SpiderMan
user-agent: SpiderView(tm)
user-agent: Spry Wizard Robot
user-agent: Site Searcher
user-agent: Suke
user-agent: suntek search engine
user-agent: Sven
user-agent: Sygol
user-agent: TACH Black Widow
user-agent: Tarantula
user-agent: tarspider
user-agent: Tcl W3 Robot
user-agent: TechBOT
user-agent: Templeton
user-agent: TeomaTechnologies
user-agent: TITAN
user-agent: TitIn
user-agent: The TkWWW Robot
user-agent: TLSpider
user-agent: UCSD Crawl
user-agent: UdmSearch
user-agent: UptimeBot
user-agent: URL Check
user-agent: URL Spider Pro
user-agent: Valkyrie
user-agent: Verticrawl
user-agent: Victoria
user-agent: vision-search
user-agent: void-bot
user-agent: Voyager
user-agent: VWbot
user-agent: The NWI Robot
user-agent: W3M2
user-agent: WallPaper (alias crawlpaper)
user-agent: the World Wide Web Wanderer
user-agent: w@pSpider by wap4.com
user-agent: WebBandit Web Spider
user-agent: WebCatcher
user-agent: WebCopy
user-agent: webfetcher
user-agent: The Webfoot Robot
user-agent: Webinator
user-agent: weblayers
user-agent: WebLinker
user-agent: WebMirror
user-agent: The Web Moose
user-agent: WebQuest
user-agent: Digimarc MarcSpider
user-agent: WebReaper
user-agent: webs
user-agent: Websnarf
user-agent: WebSpider
user-agent: WebVac
user-agent: webwalk
user-agent: WebWalker
user-agent: WebWatch
user-agent: Wget
user-agent: whatUseek Winona
user-agent: WhoWhere Robot
user-agent: Wired Digital
user-agent: Weblog Monitor
user-agent: w3mir
user-agent: WebStolperer
user-agent: The Web Wombat
user-agent: The World Wide Web Worm
user-agent: WWWC Ver 0.2.5
user-agent: WebZinger
user-agent: XGET
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: Meltwater
Disallow: /
User-agent: Digimind
Disallow: /
User-agent: Knowings
Disallow: /
User-agent: kbcrawl
Disallow: /
User-agent: AmiSoftware
Disallow: /
User-agent: Ask n read
Disallow: /
User-agent: Qwam content intelligence
Disallow: /
User-agent: Zite
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Youmag
Disallow: /
User-agent: Synthesio
Disallow: /
User-agent: trendybuzz
Disallow: /
User-agent: spotter
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: 5emeRue
Disallow: /
User-agent: Augure
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: Corporama
Disallow: /
User-agent: readability.com
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: libwww
Disallow: /
User-agent: 5erue
Disallow: /
User-agent: adequat
Disallow: /
User-agent: adequat-systems
Disallow: /
User-agent: auramundi
Disallow: /
User-agent: coexel
Disallow: /
User-agent: leadbox
Disallow: /
User-agent: Moreover
Disallow: /
User-agent: mytwip
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: opinion-tracker
Disallow: /
User-agent: proxem
Disallow: /
User-agent: score3
Disallow: /
User-agent: trendeo
Disallow: /
User-agent: vecteurplus
Disallow: /
User-agent: verticalsearch
Disallow: /
User-agent: vsw
Disallow: /
User-agent: winello
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: infoseek
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: asknread.com
Disallow: /
User-agent: ellisphere
Disallow: /
User-agent: spotter
Disallow: /
User-agent: 008
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: AlvinetSpider
Disallow: /
User-agent: Antenne Hatena
Disallow: /
User-agent: ApocalXExplorerBot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: BizInformation
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: DISCo Pump 3.1
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: DotBot/1.1
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTrack 3.0
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: Igentia
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: InfoPath
Disallow: /
User-agent: InfoPath.3
Disallow: /
User-agent: Cyberwatcher
Disallow: /
User-agent: SafeDNSBot (https://www.safedns.com/searchbot)
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: Naverbot
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetAttache
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: OpenindexSpider
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RMA
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SightupBot
Disallow: /
User-agent: SiteBot
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: SiteSucker
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: sosospider
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: suggybot
Disallow: /
User-agent: SuperBot
Disallow: /
User-agent: SuperBot/2.6
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: TightTwatBot
Disallow: /
User-agent: Titan
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: TosCrawler
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: True_Robot/1.0
Disallow: /
User-agent: turingos
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: UrlPouls
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: VCI
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebBandit/3.50
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: webcopy
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: webmirror
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: website extractor
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebStripper/2.02
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: wget
Disallow: /
User-agent: WikioFeedBot
Disallow: /
User-agent: WinHTTrack
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: yacy
Disallow: /
User-agent: YRSPider
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zookabot
Disallow: /
## EXCLUSION BOTS
User-agent: ConveraCrawler
User-agent: Jetbot
User-agent: Meltwater
User-agent: Digimind
User-agent: Knowings
User-agent: kbcrawl
User-agent: AmiSoftware
User-agent: Ask n read
User-agent: Qwam content intelligence
User-agent: Zite
User-agent: FlipboardProxy
User-agent: Youmag
User-agent: Synthesio
User-agent: trendybuzz
User-agent: spotter
User-agent: linkfluence
User-agent: 5emeRue
User-agent: Augure
User-agent: Sindup
User-agent: Corporama
User-agent: readability.com
User-agent: grub-client
User-agent: ia_archiver
User-agent: ia_archiver-web.archive.org
User-agent: k2spider
User-agent: libwww
User-agent: 5erue
User-agent: adequat
User-agent: adequat-systems
User-agent: auramundi
User-agent: coexel
User-agent: leadbox
User-agent: Moreover
User-agent: mytwip
User-agent: NewsNow
User-agent: Newzbin
User-agent: opinion-tracker
User-agent: proxem
User-agent: score3
User-agent: trendeo
User-agent: vecteurplus
User-agent: verticalsearch
User-agent: vsw
User-agent: winello
User-agent: Fetch
User-agent: infoseek
User-agent: MSIECrawler
User-agent: sitecheck.internetseer.com
User-agent: TeleportPro
User-agent: Zealbot
User-agent: asknread.com
User-agent: ellisphere
User-agent: spotter
User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: yacy
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Linguee Bot
Disallow: /
User-agent: BLP_bbot/0.1
Disallow: /
User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: YandexSomething/1.0
Disallow: /
User-agent: Java/1.6.0_04
Disallow: /
User-agent: 8484 Boston Project v 1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; m)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; H)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; P)
Disallow: /
User-agent: Yandex/1.01.001 (compatible; Win16; I)
Disallow: /
User-agent: +BecomeBot/3.0
Disallow: /
User-agent: AISearchBot
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: AmazonCloudServices
Disallow: /
User-agent: Apexoo Spider 1.0
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: Arachnoidea
Disallow: /
User-agent: ArchitextSpider
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: DotBot/1.1
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
User-agent: SemrushBot-SA/0.97
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Disallow: /
User-agent: AsapBot
Disallow: /
User-agent: semantic-visions.com crawler
Disallow: /
User-agent: MetaURI API/2.0 +metauri.com
Disallow: /
User-agent: MetaURI
Disallow: /
User-agent: citeseerxbot
Disallow: /
User-agent: autokrawl
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /
User-agent: becomebot
Disallow: /
User-agent: BecomeBot/3.0
Disallow: /
User-agent: Bitacle
Disallow: /
User-agent: Bitacle bot
Disallow: /
User-agent: Bitacle bot/1.1
Disallow: /
User-agent: Bitacle*
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: BobCrawl
Disallow: /
User-agent: boitho
Disallow: /
User-agent: boitho.com
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: bot/1.0
Disallow: /
User-agent: bot/1.0+(bot;+http://;+bot@bot.bot)
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: botmobi
Disallow: /
User-agent: bsalsa
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Butterfly
Disallow: /
User-agent: Cazoodle-Bot
Disallow: /
User-agent: CazoodleBot
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: Charlotte
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: classroombot
Disallow: /
User-agent: ConveraCrawler/0.9d
Disallow: /
User-agent: ConveraCrawler/0.9e
Disallow: /
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: crawl13.exabot.com
Disallow: /
User-agent: crawl6.exabot.com
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: cyberpatrol
Disallow: /
User-agent: cyberpatrolcrawler
Disallow: /
User-agent: datacha0s/2.0
Disallow: /
User-agent: DBLBot
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
Disallow: /
User-agent: dcbot
Disallow: /
User-agent: del.icio.us
Disallow: /
User-agent: del.icio.us-thumbnails/1.0
Disallow: /
User-agent: DepSpid
Disallow: /
User-agent: discobot
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: DTAAgent
Disallow: /
User-agent: eagle
Disallow: /
User-agent: ee:
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: envolk
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: EUDORA
Disallow: /
User-agent: Everest-Vulcan
Disallow: /
User-agent: exabot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Exabot-Images/1.0
Disallow: /
User-agent: Exabot-Images/3.0
Disallow: /
User-agent: Exabot/2.0
Disallow: /
User-agent: Exabot/3.0
Disallow: /
User-agent: exooba
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: FairShare
Disallow: /
User-agent: fast
Disallow: /
User-agent: fast Enterprise crawler
Disallow: /
User-agent: fast metaweb crawler
Disallow: /
User-agent: FAST-WebCrawler
Disallow: /
User-agent: fast-Webcrawler
Disallow: /
User-agent: findlinks/1.1.1-a5
Disallow: /
User-agent: findlinks/1.1.3-beta9
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: flatlandbot
Disallow: /
User-agent: jack
Disallow: /
User-agent: JS-Kit URL Resolver
Disallow: /
User-agent: KomodiaBot
Disallow: /
User-agent: FollowSiteBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: FunWebProducts
Disallow: /
User-agent: FunWebProducts f39-2350
Disallow: /
User-agent: FyberSpider
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: genieBot
Disallow: /
User-agent: genieBot (wgao@genieknows.com)
Disallow: /
User-agent: genieBot+(wgao@genieknows.com)
Disallow: /
User-agent: genieBot+wgao@genieknows.com
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Gigabot*
Disallow: /
User-agent: Gigabot/2.0
Disallow: /
User-agent: GingerCrawler
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: HenryTheMiragoRobot
Disallow: /
User-agent: heritrix
Disallow: /
User-agent: heritrix/1.12.1 +http://www.page-store.com)
Disallow: /
User-agent: hloader
Disallow: /
User-agent: HMSE_Robot
Disallow: /
User-agent: HornySexSearch
Disallow: /
User-agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /
User-agent: http://www.cuill.com/twiceler/robot.html
Disallow: /
User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/)
Disallow: /
User-agent: httplib
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archive
Disallow: /
User-agent: ia_archiver-web.archive.org
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: IBM
Disallow: /
User-agent: IBM_Planetwide
Disallow: /
User-agent: ichiro/2.0
Disallow: /
User-agent: ichiro/2.0 (ichiro@nttr.co.jp)
Disallow: /
User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
Disallow: /
User-agent: IconSurf
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: InfoPath
Disallow: /
User-agent: InfoPath.1
Disallow: /
User-agent: InfoPath.2
Disallow: /
User-agent: IntegraTelecom
Disallow: /
User-agent: internetseer
Disallow: /
User-agent: InternetSeer.com
Disallow: /
User-agent: ips-agent
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: IRLbot/3.0
Disallow: /
User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: Isidorus
Disallow: /
User-agent: Java Bot 2.0
Disallow: /
User-agent: Java/1.5.0_05
Disallow: /
User-agent: Java/1.5.0_10
Disallow: /
User-agent: JavaBot
Disallow: /
User-agent: JavaBot 2.0
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Jyxobot
Disallow: /
User-agent: kalooga
Disallow: /
User-agent: KaloogaBot
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Disallow: /
User-agent: kyluka
Disallow: /
User-agent: LaBot
Disallow: /
User-agent: larbin
Disallow: /
User-agent: larbin_2.6.1
Disallow: /
User-agent: larbin_2.6.1+(larbin2.6.1@unspecified.mail)
Disallow: /
User-agent: laycat
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: LexiBotWeb Image Collector
Disallow: /
User-agent: libcurl-agent
Disallow: /
User-agent: LinkAider
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkWalker/2.0
Disallow: /
User-agent: LinkWalker/2.0 - www.seventwentyfour.com
Disallow: /
User-agent: LMQueueBot/0.2
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: LocalBot
Disallow: /
User-agent: LocalcomBot
Disallow: /
User-agent: looksmart
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: MantraAgent
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MJ12bot/v1.2.0
Disallow: /
User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? )
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: Modified GMMBotRightHere
Disallow: /
User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Disallow: /
User-agent: Mozilla/5.0+(compatible;+BecomeBot/3.0;++http://www.become.com/site_owners.html)
Disallow: /
User-agent: MSR-ISRCCrawler
Disallow: /
User-agent: nambu
Disallow: /
User-agent: NationalDirectory-SuperSpider
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: nexen
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: NG/2.0
Disallow: /
User-agent: nicebot
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: noxtrumbot/1.0
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: obot
Disallow: /
User-agent: Ocelli
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Offline Explorer/1.5
Disallow: /
User-agent: Offline Explorer/1.9
Disallow: /
User-agent: Offline Explorer/2.5
Disallow: /
User-agent: OmniExplorer
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: OmniExplorer_Bot/1.07
Disallow: /
User-agent: OmniExplorer_Bot/1.09
Disallow: /
User-agent: OmniExplorer_Bot/1.10
Disallow: /
User-agent: OnTownsBot
Disallow: /
User-agent: OOZBOT
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind data gathere
Disallow: /
User-agent: OpenX
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: page_prefetcher
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: Piffany_Web_Cacher_v0.91
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: psbot/0.1
Disallow: /
User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html)
Disallow: /
User-agent: PSIBots
Disallow: /
User-agent: QEAVis
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: Radiation Retriever 1.1
Disallow: /
User-agent: Rankivabot
Disallow: /
User-agent: REAP-Crawler
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-agent: RiceComputerArchitecture
Disallow: /
User-agent: RMA
Disallow: /
User-agent: robotgenius
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: ru_com_viewer
Disallow: /
User-agent: RufusBot
Disallow: /
User-agent: SapphireWebCrawler
Disallow: /
User-agent: Twiceler www.cuill.com/robots.html
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: wget
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: IsraBot
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: fast
Disallow: /
User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Disallow: /
User-agent: Twiceler www.cuill.com/twiceler/robot.html
Disallow: /
User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Disallow: /
User-agent: sbider
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: SBIder/SBIder-0.8.2-dev
Disallow: /
User-agent: scooter
Disallow: /
User-agent: Scooter/2.0
Disallow: /
User-agent: scoutjet
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: searchme
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: Sensis Web Crawler
Disallow: /
User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)
Disallow: /
User-agent: Sensis+Web+Crawler+
Disallow: /
User-agent: Servage
Disallow: /
User-agent: ServageRobot
Disallow: /
User-agent: setooz
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: Sidewinder
Disallow: /
User-agent: Sika
Disallow: /
User-agent: sitecat
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: Slicehost
Disallow: /
User-agent: Slurp.so/1.0
Disallow: /
User-agent: slurp@inktomi
Disallow: /
User-agent: slurp@inktomi.com
Disallow: /
User-agent: Slurp/2.0-KiteHourly
Disallow: /
User-agent: Slurp/2.0-OwlWeekly
Disallow: /
User-agent: Slurp/2.0j
Disallow: /
User-agent: Slurp/3.0-AU
Disallow: /
User-agent: SlySearch
Disallow: /
User-agent: SlySearch/1.0
Disallow: /
User-agent: SlySearch/1.x
Disallow: /
User-agent: Snapbot
Disallow: /
User-agent: Snapbot/1.0
Disallow: /
User-agent: SnapPreviewBot
Disallow: /
User-agent: Snappy
Disallow: /
User-agent: Snappybot
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: sogou+spider
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: Speedy
Disallow: /
User-agent: speedyspider
Disallow: /
User-agent: sphsearch
Disallow: /
User-agent: sproose
Disallow: /
User-agent: StackRambler/2.0
Disallow: /
User-agent: StackRambler/2.0+(MSIE+incompatible)
Disallow: /
User-agent: SuperBot/4.4.0.60 (Windows XP)
Disallow: /
User-agent: SuperPages
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: SurveyBot/2.3
Disallow: /
User-agent: SurveyBot/2.3 (Whois Source)
Disallow: /
User-agent: uberbot
Disallow: /
User-agent: UltraSeek
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
Disallow: /
User-agent: ViolaBot
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebCopier v3.2a
Disallow: /
User-agent: WebCopier v3.3
Disallow: /
User-agent: WebCopier v3.3.0
Disallow: /
User-agent: WebCopier v3.3.2
Disallow: /
User-agent: WebCrawler
Disallow: /
User-agent: WebDataCentreBot
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebmasterWorld Extractor
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: Websense
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webspider
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebViewer
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: WebZIP/3.5 (http://www.spidersoft.com)
Disallow: /
User-agent: whois.de
Disallow: /
User-agent: whoisde.de
Disallow: /
User-agent: WillyBot
Disallow: /
User-agent: Win32
Disallow: /
User-agent: worio
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Xenu Link Sleuth 1.2i
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: Xenu Link Sleuth/1.3.8
Disallow: /
User-agent: Y!J
Disallow: /
User-agent: y!j-bsc
Disallow: /
User-agent: Y!J-BSC
Disallow: /
User-agent: Y!J-BSC/1.0
Disallow: /
User-agent: TunitinBot
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: Meltwater
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Digimind
Disallow: /
User-agent: Knowings D
Disallow: /
User-agent: kbcrawl
Disallow: /
User-agent: Wget
Disallow: /
User-agent: AmiSoftware
Disallow: /
User-agent: Newzbin
Disallow: /
User-agent: Ask n read
Disallow: /
User-agent: Qwam content intelligence
Disallow: /
User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Disallow: /
User-agent: y!j/1.0
Disallow: /
User-agent: Yahoo-Blogs
Disallow: /
User-agent: Yahoo-Blogs/v3.9
Disallow: /
User-agent: Yahoo-MMCrawler/3.x
Disallow: /
User-agent: YahooSeeker
Disallow: /
User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
Disallow: /
User-agent: YahooYSMcm
Disallow: /
User-agent: Zite
Disallow: /
User-agent: flipboard
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Youmag
Disallow: /
User-agent: YahooYSMcm/2.0.0
Disallow: /
User-agent: yandex
Disallow: /
User-agent: YandexSomething
Disallow: /
User-agent: Yanga
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: yoogliFetchAgent/0.1
Disallow: /
User-agent: Zao
Disallow: /
User-agent: zermelo
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: zibber
Disallow: /
User-agent: zibber v0.1
Disallow: /
User-agent: zibber-v0.1(www.zibb.com/crawler/)
Disallow: /
User-agent: ZupeeCrawler
Disallow: /
User-agent: zyborg
Disallow: /
User-agent: ZyBorg/1.0
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Snoothbot
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Daum
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: WeSEE
Disallow: /
User-agent: ShopWiki
Disallow: /
user-agent: crystalsemantics
disallow: /
user-agent: f-six
disallow: /
user-agent: sitebot
disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: scrapybot
Disallow: /
User-agent: VocusBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: ezooms.bot
Disallow: /
User-agent: skimlinks.com
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: wikiwix-bot
Disallow: /
User-agent: semantissimo
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: LemonSources
Disallow: /
User-agent: discobot
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: ltbot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: language-tools.com
Disallow: /
User-agent: TREC-KBA-Bot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: Socialradarbot
Disallow: /
User-agent: Havij
Disallow: /
User-agent: proximic
Disallow: /
User-agent: MireoBot
Disallow: /
User-agent: brandwatch
Disallow: /
User-agent: seoprofiler
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Bot clicker
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: EC2LinkFinder
Disallow: /
User-agent: ExaleadCloudview
Disallow: /
User-agent: AdMedia
Disallow: /
User-agent: grepnetstat.com
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: NerdByNature
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: netseer
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: netseer.com
Disallow: /
User-agent: nagios
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: FeedRetriever
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: Argus Presse
Disallow: /
User-agent: IBM ICA Crawler
Disallow: /
User-agent: nutch agent
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: PaperLiBot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: ip-web-crawler.com
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: LinksCrawler
Disallow: /
User-agent: archive.org
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: netEstate
Disallow: /
User-agent: MyNutchTest
Disallow: /
User-agent: admantx
Disallow: /
User-agent: genieo.com
Disallow: /
User-agent: mixdata.com
Disallow: /
User-agent: ubermetrics-technologies
Disallow: /
User-agent: grapeshot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: WASALive
Disallow: /
User-agent: URLAppendBot
Disallow: /
User-agent: MaxPointCrawler
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Aboundexbot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: psbot
Disallow: /
User-Agent: Slurp
User-agent: Yahoo! Slurp
User-agent: Baiduspider
User-agent: TweetedTimes Bot
User-agent: TweetmemeBot
User-agent: PaperLiBot
User-agent: BomboraBot
User-agent: WF search
User-agent: heritrix
User-agent: expo9
User-agent: NetSeer
User-agent: AddThis.com
User-agent: Feed
User-agent: Sogou
User-agent: ZumBot
User-agent: Feedspot
User-agent: acebookexternalhit
User-agent: StorygizeBot
User-agent: Scrapy
User-agent: GermCrawler
User-agent: Netvibes
User-agent: Vagabondo
User-agent: Genieo
User-agent: um-IC
User-agent: Typhoeus
User-agent: newspaper
User-agent: weborama-fetcher
User-agent: ScooperBot
User-agent: BYO-2
User-agent: SimplePie
User-agent: GroupHigh
User-agent: feedparser
User-agent: UniversalFeedParser
User-agent: Pulsepoint
User-agent: Steeler
User-agent: GarlikCrawler
User-agent: Flipboard
User-agent: FlipboardRSS
User-agent: Moreover
User-agent: Mercury
User-agent: FeedBurner
User-agent: Yahoo
User-agent: omgili
User-agent: Bloglovin
User-agent: YahooCacheSystem
User-agent: mTalk
User-agent: EveryoneSocialBot
User-agent: EzineArticlesLinkScanner
User-agent: dlvr.it
User-agent: pfblogs
User-agent: Aol
User-agent: ToutiaoSpider
User-agent: Photon
User-agent: PicoFeed
User-agent: um-FC
User-agent: Digg
User-agent: Synaptive
User-agent: YisouSpider
User-agent: BYO-4
User-agent: Feedbin
User-agent: Twingly
User-agent: Tiny
User-agent: WWW-Mechanize
User-agent: CommaFeed
User-agent: TkBot
User-agent: TinEye-bot
User-agent: Exabot
User-agent: coccoc
User-agent: BLEXBot
User-agent: uMBot-LN
User-agent: QuerySeekerSpider
User-agent: Yandex
User-agent: YandexBot
User-agent: YandexMobileBot
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: sogou spider
User-agent: Sogou web spider
User-agent: Baiduspider
User-agent: YoudaoBot
User-agent: MJ12bot
User-agent: spbot
User-agent: 360Spider
User-agent: SeznamBot
User-Agent: AhrefsBot
User-Agent: Linkdex
User-agent: linkdexbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: CFNetwork
User-agent: SputnikBot
User-agent: TurnitinBot
User-agent: proximic
User-agent: grapeshot
User-agent: VigLink
User-agent: Clickagy
User-agent: YaK
User-agent: DeuSu
User-agent: Taboolabot
User-agent: Pinterestbot
User-agent: psbot
User-agent: Plukkie
User-agent: GetIntent
User-agent: Facebot
User-agent: linkdexbot
User-agent: Vegi
User-agent: istellabot
User-agent: Leikibot
User-agent: Claritybot
User-agent: PiplBot
User-agent: Feedspotbot
User-agent: moatbot
User-agent: Yeti
User-agent: contxbot
User-agent: SEOkicks-Robot
User-agent: rogerbot
User-agent: Discordbot
User-agent: Tapatalk
User-agent: Mail.RU_Bot
User-agent: coccocbot
User-agent: HybridBot
User-agent: Slack-ImgProxy
User-agent: adbeat_bot
User-agent: Kraken
User-agent: MojeekBot
User-agent: calculon
User-agent: mcsbot
User-agent: avkzarabotok
User-agent: SearchmetricsBot
User-agent: Uptimebot
User-agent: yacybot
User-agent: Applebot
User-agent: Gigabot
User-agent: g2reader-bot
User-agent: Seoterritory
User-agent: AhrefsBot
User-agent: feedbot
User-agent: Findxbot
User-agent: Wotbox
User-agent: SMTBot
User-agent: Ocarinabot
User-agent: Dragonbot
User-agent: Zombiebot
User-agent: metabot.ru
User-agent: RSSMicro
User-agent: _zbot
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Cliqzbot
User-agent: DotBot
User-agent: Exabot
User-agent: moatbot
User-agent: Tapatalk
Disallow: /
User-agent: GingerCrawler
Disallow: /
User-agent: nutch
Disallow: /
User-agent: Urlbot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: 008
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Speedy Spider
Disallow: /
User-agent: Accelobot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: discobot
Disallow: /
User-agent: AboutUsBot
Disallow: /
User-agent: Vocus
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: http://ahrefs.com/robot
Disallow: /
User-agent: http://www.genieo.com/webfilter.html
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: linkdex.com/v2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Disallow: /
User-agent: NetcraftSurveyAgent/1.0; +info@netcraft.com
Disallow: /
User-agent: NetcraftSurveyAgent/1.0
Disallow: /
User-agent: NetcraftSurveyAgent
Disallow: /
User-agent: NetcraftSurveyAgent/2.0
Disallow: /
User-agent: SemrushBot/0.95
Disallow: /
User-agent: MJ12bot/v1.4.3
Disallow: /
User-agent: Java/1.6.0_04
Disallow: /
User-agent: JS-Kit URL Resolver, http://js-kit.com/
Disallow: /
User-agent: JS-Kit
Disallow: /
User-agent: NetShelter ContentScan
Disallow: /
User-agent: Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Disallow: /
User-agent: Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Disallow: /
User-agent: Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Disallow: /
User-agent: YYSpider
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: PagePeeker2
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: Abonti
Disallow: /
User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: Image\ Sucker
Disallow: /
User-agent: SuperBot
Disallow: /
User-agent: Web\ Image\ Collector
Disallow: /
User-agent: Web\ Sucker
Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: linko
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
США - 173.194.71.132
HTTP/1.1 200 OK
Date: Sun, 07 Oct 2018 11:57:37 GMT
Server: Apache/2.4.18 (Ubuntu)
Set-Cookie: PHPSESSID=kopevj7362437m8vegsjbf26v4; path=/; domain=.raetsel-hilfe.de; secure; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Content-Language: de
Кнопка для анализа сайта в один клик, для установки перетащите ссылку на "Панель закладок"