Возраст домена | 15 лет |
Дата окончания | Истек срок регистрации |
PR | 1 |
ИКС | n/a |
Страниц в Google | 2 |
Страниц в Яндексе | n/a |
Dmoz | Нет |
Яндекс Каталог | Нет |
Alexa Traffic Rank | Нет данных |
Alexa Country | Нет данных |
История изменения показателей | Авторизация |
Идет сбор информации... Обновить
www.jaguar-zelja.com - Naslovnica
Radio za dusu
Radio za dusu
UTF-8
12.16 КБ
176
1 516 симв.
1 290 симв.
Данные linkpad ( 17 Апреля 2016 ) | |
Количество ссылок на сайт | 22 |
Количество доменов, которые ссылаются на сайт | 5 |
Количество найденных анкоров | 5 |
Исходящие (внешние) ссылки домена | 54 |
Количество доменов, на которые ссылается сайт | 14 |
Количество исходящих анкоров | 11 |
Внутренние ссылки главной страницы ( 6 ) | |
# | - |
jaguar-zelja.com/index.php | Naslovnica |
/index.php/component/tags/tag/jagi.html | Jagi |
/index.php/component/users/?view=reset&Itemid=101 | Zaboravili ste lozinku? |
/index.php/component/users/?view=remind&Itemid=101 | Zaboravili ste korisničko ime? |
/index.php/component/users/?view=registration&Itemid=101 | Kreiraj korisnički račun |
Domain names in the .com and .net domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.
Domain Name: JAGUAR-ZELJA.COM
Registrar: 1 API GMBH
Sponsoring Registrar IANA ID: 1387
Whois Server: whois.1api.net
Referral URL: http://www.1api.net
Name Server: NS1.HANS.HOSTEUROPE.DE
Name Server: NS2.HANS.HOSTEUROPE.DE
Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Updated Date: 07-jan-2016
Creation Date: 07-jan-2009
Expiration Date: 07-jan-2017
>>> Last update of whois database: Wed, 02 Nov 2016 18:16:46 GMT <<<
For more information on Whois status codes, please visit https://icann.org/epp
NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar. Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.
TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability. VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.
The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
Domain Name: JAGUAR-ZELJA.COM
Registry Domain ID: 1536343932_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.1api.net
Registrar URL: http://www.1api.net
Updated Date: 2014-08-18T11:45:20Z
Creation Date: 2009-01-07T01:11:22Z
Registrar Registration Expiration Date: 2017-01-07T01:11:36Z
Registrar: 1API GmbH
Registrar IANA ID: 1387
Registrar Abuse Contact Email: abuse@1api.net
Registrar Abuse Contact Phone: +49.68416984x200
Reseller: dynamic-net.ch AG www.dynamic-net.ch
Domain Status: clientTransferProhibited - http://www.icann.org/epp#clientTransferProhibited
Registry Registrant ID:
Registrant Name: Jurakic Mato
Registrant Organization:
Registrant Street: Neudorfstrasse+6
Registrant City: Rohr
Registrant State/Province:
Registrant Postal Code: 5032
Registrant Country: CH
Registrant Phone: +41.628226726
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: cigo@hispeed.ch
Registry Admin ID:
Admin Name: Jurakic Mato
Admin Organization:
Admin Street: Neudorfstrasse+6
Admin City: Rohr
Admin State/Province:
Admin Postal Code: 5032
Admin Country: CH
Admin Phone: +41.628226726
Admin Phone Ext:
Admin Fax:
Admin Fax Ext:
Admin Email: cigo@hispeed.ch
Registry Tech ID:
Tech Name: Hosteurope Hostmaster
Tech Organization: Host Europe Suisse AG
Tech Street: TECHNOPARK LUZERN, D4 Platz 4
Tech City: Root Laengenbold
Tech State/Province:
Tech Postal Code: 6039
Tech Country: CH
Tech Phone: +41.414555910
Tech Phone Ext:
Tech Fax: +41.414555949
Tech Fax Ext:
Tech Email: admin@hosteurope.ch
Name Server: ns1.hans.hosteurope.de
Name Server: ns2.hans.hosteurope.de
http://wdprs.internic.net/
>>> Last update of WHOIS database: 2016-11-02T18:17:04Z <<<
For more information on Whois status codes, please visit https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en.
; This data is provided for information purposes, and to assist persons
; obtaining information about or related to domain name registration
; records. We do not guarantee its accuracy.
; By submitting a WHOIS query, you agree that you will use this data
; only for lawful purposes and that, under no circumstances, you will
; use this data to
; 1) allow, enable, or otherwise support the transmission of mass
; unsolicited, commercial advertising or solicitations via E-mail
; (spam); or
; 2) enable high volume, automated, electronic processes that apply
; to this WHOIS server.
; These terms may be changed without prior notice.
; By submitting this query, you agree to abide by this policy.
; Domain managed by dynamic-net.ch AG
User-agent: *
Disallow: /
#User-agent: *
#Disallow: /var/www/vhosts/example.com/httpdocs/
User-agent: Mail.RU_Bot
# oder
User-agent: Mail.RU
Disallow: / cgi-bin
Disallow: / add
Allow: / add / top
Disallow: / add / top / private
User-agent: OutfoxBot/0.5
User-agent: complex_network_group
User-agent: imparser09.yandex.ru
User-agent: spider14.yandex.ru
User-agent: yandex.ru
User-agent: yandex
User-agent: Alexibot
User-agent: SemrushBot
Disallow: /
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: BackDoorBot/1.0
User-agent: BPImageWalker/2.0
User-agent: Black.Hole
User-agent: BlackWidow
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: Bot mailto:craftbot@yahoo.com
User-agent: BotALot
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: DISCo
User-agent: DISCo Pump 3.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: DittoSpyder
User-agent: Download Demon
User-agent: Download Demon/3.2.0.8
User-agent: Download Demon/3.5.0.11
User-agent: EirGrabber
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Express WebPictures
User-agent: Express WebPictures (www.express-soft.com)
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: Flaming AttackBot
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: Foobot
User-agent: FrontPage
User-agent: FrontPage [NC,OR]
User-agent: Gaisbot
User-agent: GetRight
User-agent: GetRight/2.11
User-agent: GetRight/3.1
User-agent: GetRight/3.2
User-agent: GetRight/3.3
User-agent: GetRight/3.3.3
User-agent: GetRight/3.3.4
User-agent: GetRight/4.0.0
User-agent: GetRight/4.1.0
User-agent: GetRight/4.1.1
User-agent: GetRight/4.1.2
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b1
User-agent: GetRight/4.5b2
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go!Zilla 3.3 (www.gozilla.com)
User-agent: Go!Zilla 3.5 (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: HTTrack 3.0x
User-agent: HTTrack [NC,OR]
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: Indy Library
User-agent: Indy Library [NC,OR]
User-agent: InfoNaviRobot
User-agent: InterGET
User-agent: Internet Ninja
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: Iron33/1.0.2
User-agent: JOC Web Spider
User-agent: JennyBot
User-agent: JetCar
User-agent: Kenjin Spider
User-agent: Kenjin.Spider
User-agent: Keyword Density/0.9
User-agent: Keyword.Density
User-agent: LNSpiderguy
User-agent: LeechFTP
User-agent: LexiBot
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: Mister PiX
User-agent: Mister PiX version.dll
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister.PiX
User-agent: NICErsPRO
User-agent: NPBot
User-agent: NPbot
User-agent: Navroad
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.10
User-agent: NetAnts/1.23
User-agent: NetAnts/1.24
User-agent: NetAnts/1.25
User-agent: NetMechanic
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.1
User-agent: Offline Explorer/2.3
User-agent: Offline Explorer/2.4
User-agent: Offline Explorer/2.5
User-agent: Offline Navigator
User-agent: Offline.Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Oracle Ultra Search
User-agent: PageGrabber
User-agent: Papa Foto
User-agent: PerMan
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QueryN.Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: ReGet
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2.0
User-agent: WebCopier
User-agent: WebCopier v.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website eXtractor
User-agent: Website eXtractor (http:/www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: Wget
User-agent: Wget/1.10.2
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1+cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmos
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb/clsHTTP
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: tAkeOut
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
# Ganze Site f�r alle Robots sperren
User-agent: *
Disallow: /
user-agent: AhrefsBot
disalow: /
User-agent: YandeG 1.02
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
User-agent: NerdyBot
Disallow: /
User-agent: *
Disallow: /report/
Disallow: /reports/
Disallow: /foren/admin/
Disallow: /foren/attach_mod/
Disallow: /foren/db/
Disallow: /foren/files/
Disallow: /foren/includes/
Disallow: /foren/language/
Disallow: /foren/spelling/
Disallow: /foren/common.php
Disallow: /foren/config.php
Disallow: /foren/groupcp.php
Disallow: /foren/memberlist/
Disallow: /foren/memberlist.php
Disallow: /foren/memberlist.html
Disallow: /foren/mini_cal.php
Disallow: /foren/modcp/
Disallow: /foren/posting/
Disallow: /foren/posting.php
Disallow: /foren/posting/mode-reply/
Disallow: /foren/posting/mode-quote/
Disallow: /foren/privmsg/
Disallow: /foren/profile/
Disallow: /foren/profile.php
Disallow: /foren/ranks.php
Disallow: /foren/search/
Disallow: /foren/search.php
Disallow: /foren/statistics.php
Disallow: /foren/viewonline.php
Disallow: /foren/login.php
Disallow: /foren/login/
Disallow: /foren/*?
Disallow: /outbound.pl
Disallow: /foren_BISHER_2_0_1/
Disallow: /foren2/
Disallow: /foren/*?printertopic=
Disallow: /?*
Disallow: /seoblog/*login.php
Disallow: /seoblog/*/comment-page*
Disallow: /cgi-bin/
#
# ACHTUNG!!!
#
# An dieser Datei KEINE Aenderungen vornehmen!!!
# Dies ist eine automatisch erstellte Datei!
# Aenderungen werden ohne Sicherung automatisch
# ueberschrieben!
#
#-------------------------------------------------
# Version: 2013-11-13 19:35
# Generiert: 13.11.2013 19:35
# durch Script: esRemoteAccessSettings.class.php
#-------------------------------------------------
# 80legs (http://www.80legs.com/webcrawler.html)
User-agent: 008
Disallow: /
# Abonti (http://www.abonti.com)
User-agent: Abonti
Disallow: /
# AcoonBot (http://www.acoon.de/robot.asp)
User-agent: Acoon
Disallow: /
User-agent: AcoonBot
Disallow: /
User-agent: client.yota.ru
Disallow: /
# Ahrefs.com (http://ahrefs.com/robot/)
user-agent: AhrefsBot
disallow: /
# alexa.com (http://www.alexa.com/site/help/webmasters)
User-agent: ia_archiver
Disallow: /
# BacklinkCrawler (http://www.backlinktest.com/crawler.html)
User-agent: BacklinkCrawler
Disallow: /
# Baiduspider (http://www.baidu.com/search/spider.htm)
User-agent: baiduspider
Disallow: /
# Birubot [Bot]
User-agent: Birubot
Disallow: /
# Bixo Labs (http://bixolabs.com)
User-agent: bixolabs
disallow: /
# BLEXBot (http://webmeup.com/crawler.html)
User-agent: BLEXBot
Disallow: /
# BotOnParade (www.bots-on-para.de/bot.html)
User-agent: BotOnParade
Disallow: /
# BUbiNG (http://law.di.unimi.it/BUbiNG.html)
User-agent: BUbiNG
Disallow: /
# CareerBot (http://www.career-x.de/bot.html)
User-agent: careerbot
Disallow: /
# discobot (http://discoveryengine.com/discobot.html)
User-agent: discobot
Disallow: /
# Exabot/3.0 (http://www.exabot.com/go/robot)
User-agent: Exabot
Disallow: /
# Gigabot (http://www.gigablast.com/spider.html)
User-agent: gigabot
Disallow: /
# Gnip [Bot] (http://www.gnip.com)
User-agent: UnwindFetchor
Disallow: /
# Huawei Symantec (http://www.huaweisymantec.com/en/IRL/spider/)
User-agent: HuaweiSymantecSpider
Disallow: /
# iCjobs Stellenangebote Jobs (www.icjobs.de/bot.htm)
User-agent: ICCrawler - iCjobs
Disallow: /
# Linguee Bot (http://www.linguee.com/bot)
User-agent: Linguee
Disallow: /
User-agent: Exabot
Crawl-delay: 10
# Majestic-12 (http://www.majestic12.co.uk/bot.php)
User-agent: MJ12bot
Disallow: /
# MojeekBot/0.2 (http://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /
# NerdByNature (http://www.nerdbynature.net/bot)
User-agent: NerdByNature.Bot
Disallow: /
# oBot/2.3.1 (http://filterdb.iss.net/crawler/)
user-agent: oBot
disallow: /
# OneRiot [Bot] (http://www.oneriot.com)
User-agent: OneRiot
disallow: /
# Plukkie (http://www.botje.com/plukkie.htm)
User-agent: plukkie
Disallow: /
# Ruby
User-agent: Ruby
disallow: /
# ScoutJet (http://www.scoutjet.com/)
User-agent: ScoutJet
Disallow: /
# Search17Bot (http://www.search17.com/bot.php)
User-agent: search17
Disallow: /
# SearchmetricsBot (http://www.searchmetrics.com/en/searchmetrics-bot/)
User-agent: SearchmetricsBot
Disallow: /
# SEOkicks-Robot (http://www.seokicks.de/robot.html)
User-agent: seokicks
Disallow: /
# seoprofiler (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Sosospider (http://help.soso.com/webspider.htm)
User-agent: Sosospider
Disallow: /
# spbot/seoprofiler.com (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
User-agent: Speedy
Disallow: /
# Spinn3r (http://spinn3r.com/robot)
User-agent: Spinn3r
Disallow: /
# The ShopWiki Crawler (http://www.shopwiki.com/wiki/Help:Bot)
User-agent: ShopWikiDisallow: /
# TurnitinBot (http://www.turnitin.com/robot/crawlerinfo.html)
User-agent: TurnitinBot
Disallow: /
# TwengaBot (http://www.twenga.com/bot.html)
User-agent: twenga2.com
Disallow: /
User-agent: twenga.com
Disallow: /
# WBSearchBot (http://www.warebay.com/bot.html)
User-agent: WBSearchBot
Disallow: /
# Yahoo! Slurp (http://help.yahoo.com/help/us/ysearch/slurp)
User-agent: Yahoo! Slurp
Disallow: /
# Yandex bot
User-agent: Yandex
Disallow:
# Yandex (http://yandex.com/bots)
User-agent: Yandex
Disallow: /
# Zookabot (http://zookabot.com)
User-agent: Zookabot
Disallow: /
#
# Erweiterungen des Shopbetreibers
#
# robots.txt fuer http://www.sofaonline24.de
#Diese Bots wollen wir sperren
User-agent: gigabot
User-agent: psbot
User-agent: Fasterfox
User-agent: vscooter
User-agent: Teleport*
User-agent: Webwhacker*
User-agent: Webzip*
User-agent: Net Attache*
User-agent: SiteSnagger*
User-agent: HTTrack*
User-agent: WebCapture*
User-agent: WebSauger*
Disallow: /
#google darf viel aber nicht alles
User-agent: Googlebot
Disallow: /export/
Disallow: /*news_detail_print
Disallow: /*order_select
Disallow: /*order_shippay
Disallow: /*order_input
Disallow: /*order_check
Disallow: /*order_ok
Disallow: /*order_wrong
Disallow: /*order_save
Disallow: /*change
Disallow: /*favorites
Disallow: /*basket_add
Disallow: /*basket_add_hidden
Disallow: /*basket_show
Disallow: /*basket_remark
Disallow: /*basket_remark_save
Disallow: /*basket_delete
Disallow: /*basket_change_quantity
Disallow: /*logout
Disallow: /*.pdf$
#inktomi ist zu gierig und darf nicht zu oft vorbeikommen
User-agent: Slurp
Crawl-Delay: 36000
Disallow: /images/
Disallow: /export/
User-agent: bingbot
Crawl-delay: 30
#hier sollten die Seiten und Verzeichnisse stehen an die keiner darf
User-agent: *
Disallow: /export/
User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout
# oder
User-agent: *
Disallow: /search
Crawl-delay: 4.5 # specifies a 4.5 second timeout
User-agent: TwengaBot
Disallow: /
# gro�e russische Suchmaschine
User-agent: Yandex
Disallow: /
# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
http://stop-spamers-and-hackers.blogspot.ch/
User-agent: *
Disallow
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml
# Ganze Site f�r alle Robots sperren
User-agent: *
Disallow: /
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
User-agent: Yandex
Disallow: / # blocks access to the entire site
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
# robots.txt
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control - 5.01.4511
Disallow:/
User-agent: Microsoft URL Control - 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/
# Robots.txt file for http://www.jaguar-zelja.com
#
# Restrictions to all bots
# ------------------------------------------------
User-agent: *
Disallow: /adredir.php?
Disallow: /biz_redir.php?
Disallow: /redirect.php?
Disallow: /admin/
Disallow: /mm-data/
Disallow: /js/
Disallow: /css/
Disallow: /img/
Disallow: /join/
Disallow: /signin?
Disallow: /timthumb.php?
Disallow: /messages?
Disallow: /restaurant/join.php?
Disallow: /restaurant/signup
Disallow: /users/flag_content.php?
Disallow: /users/biz_sendToFriend.php?
Disallow: /faq.php
Disallow: /restaurant/manage_reviews.php
Disallow: /writeareview/restaurants
#
# Image Bot Blocks
# ------------------------------------------------
User-agent: googlebot-image
User-agent: psbot
User-agent: Yahoo-MMCrawler
Disallow: /
#
# Known Bots that stress the site
# ------------------------------------------------
# 070130 - Majestic12 Distributed Search Engine denied due to load/security
User-Agent: MJ12bot
Disallow: /
User-agent: ia_archiver
User-agent: ia_archiver*
User-agent: Linknzbot 2004
User-agent: Linknzbot*
User-agent: Linknzbot
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft URL Control - 6.01.9782
User-agent: Microsoft URL Control*
User-agent: Microsoft URL Control
User-agent: aipbot/1.0
User-agent: aipbot*
User-agent: aipbot
User-agent: LWP*
User-agent: LWP
User-agent: NutchCVS/0.06-dev
User-agent: Nutch*
User-agent: Nutch
Disallow: /
#
# General Bad Bots or Site scrapers that are completely disallowed
# ------------------------------------------------
User-agent: Alexibot
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: Black Hole
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: CherryPickerElite/1.0
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: Download Demon/3.2.0.8
User-agent: dumbot
User-agent: eCatch/3.0
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: FrontPage [NC,OR]
User-agent: GetRight/3.1
User-agent: GetRight/3.3
User-agent: GetRight/4.1.0
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: grub
User-agent: grub-client
User-agent: Harvest
User-agent: HMView
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack [NC,OR]
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: InfoNaviRobot
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: JetCar
User-agent: JOC Web Spider
User-agent: Kenjin Spider
User-agent: Keyword.Density
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: libWeb/clsHTTPDisallow: /
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a Unix Disallow: /
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LNSpiderguy
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister PiX version.dll
User-agent: Mister.PiX
User-agent: moget
User-agent: moget/2.1
User-agent: MSIECrawler
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.25
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http://www.innerprise.net/usp-spider.asp)
User-agent: NICErsPRO
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.4
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gathere
User-agent: pavuk
User-agent: pcBrowser
User-agent: Python-urllib
User-agent: Radiation Retriever 1.1
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: ReGet
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RMA
User-agent: searchpreview
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: sootle
User-agent: SpankBot
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: toCrawl/UrlDispatcher
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URLy Warning
User-agent: VoidEYE
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: WebCopier v2.5
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebmasterWorldForumBot
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: WebStripper/2.13
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebZIP/2.75 (http://www.spidersoft.com)
User-agent: WebZIP/3.65 (http://www.spidersoft.com)
User-agent: WebZIP/3.80 (http://www.spidersoft.com)
User-agent: WebZip/4.0
User-agent: WebZIP/4.0 (http://www.spidersoft.com)
User-agent: WebZIP/4.1 (http://www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http://www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http://www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http://www.spidersoft.com)
User-agent: Wget
User-agent: Wget/1.5.2
User-agent: Wget/1.7
User-agent: Wget/1.8.2
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: 008
User-agent: HttpTool*
User-agent: RealWebsiteWorth
User-agent: http://www.almaden.ibm.com/cs/crawler
User-agent: Bilbo/2.3b-UNIX
User-agent: FAST-WebCrawler/3.6/FirstPage
User-agent: Missigua Locator 1.9
User-agent: Yandex
User-Agent: OmniExplorer_Bot
User-Agent: MJ12bot
User-agent: *libwww*
User-agent: *Fetch*
User-agent: BorderManager*
User-agent: Maxthon
User-agent: Java*
User-agent: msrbot
User-agent: BSpider
User-agent: legs
User-agent: exabot
Disallow: /
# Alle Robots ausschlie�en
User-agent: *
Disallow: /temp/
User-agent: Baiduspider
Disallow: /my-page.html
User-Agent: Googlebot-Mobile
Disallow: /
User-agent: Googlebot
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: Googlebot-Mobile
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: Feedfetcher-Google
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: bingbot
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: Yahoo!
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
# alle nicht genannten Bots sperren
User-agent: *
Disallow: /
### Automatically banned scanners and crawlers section ###
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
### Section end ###
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
#Baiduspider
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider+
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
Disallow: /
User-agent: Baiduspider/2.0;+http://www.baidu.com/search/spider.html
Disallow: /
User-agent: Baiduspider/2.0
Disallow: /
User-agent: +Baiduspider
Disallow: /
User-agent: +Baiduspider/2.0
Disallow: /
User-agent: +Baiduspider/2.0;++http://www.baidu.com/search/spider.html
Disallow: /
User-agent: Mozilla/5.0(compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Disallow: /
#Yandex
User-agent: spider
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
#Sosospider
User-agent: Sosospider+
Disallow: /
#Ezooms
User-agent: ezooms
Disallow: /
User-agent: Ezooms/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot[at]gmail[dot]com)
Disallow: /
#Sogou
User-agent: sogou spider
Disallow: /
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
# robots.txt
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control - 5.01.4511
Disallow:/
User-agent: Microsoft URL Control - 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /404_files/
Disallow: /logs/
Disallow: /plesk-stat/
Disallow: /test/
User-Agent: Googlebot-Mobile
Disallow: /
User-agent: baiduSpider
Disallow: /
User-agent: BaiduSpider+
Disallow: /
User-agent: BaiduSpider+ (+; http://www.baidu.com/search/spider.htm; Ubuntu; Compatible; Version; Platform; .NET CLR 2.0.50727; Version)
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /
User-agent: Fatbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml)
Disallow: /
User-agent: genieBot
Disallow: /
User-agent: genieBot (http://64.5.245.11/faq/faq.html)
Disallow: /
User-agent: genieBot enash@genieknows.com
Disallow: /
User-agent: geniebot wgao@genieknows.com
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Gigabot/1.0
Disallow: /
User-agent: Gigabot/2.0
Disallow: /
User-agent: Gigabot/2.0 (http://www.gigablast.com/spider.html)
Disallow: /
User-agent: Gigabot/2.0/gigablast.com/spider.html
Disallow: /
User-agent: Gigabot/2.0; http://www.gigablast.com/spider.html
Disallow: /
User-agent: Gigabot/2.0att
Disallow: /
User-agent: Gigabot/3.0 (http://www.gigablast.com/spider.html)
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MJ12bot/v1.0.0 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.1 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.2 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.3 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.4 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.5 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.6 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.1.0 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer
Disallow: /
User-agent: OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com)
Disallow: /
User-agent: OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Cars Crawler
Disallow: /
User-agent: OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Internet Indexer
Disallow: /
User-agent: OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com)
Disallow: /
User-agent: OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com) Jobs Crawler
Disallow: /
User-agent: OmniExplorer_Bot/2.47 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/2.93 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.06c (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.06d (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.10 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.10a (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.11a (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.11c (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.20b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.21 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.23 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.24 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.26b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.26c (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.28 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.94b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.95 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.98b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.99 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/5.25 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/5.73 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/5.77e (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.0 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.13 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.7 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.9 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.70 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
Disallow: /
User-agent: SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
Disallow: /
User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Disallow: /
User-agent: WebAlta Crawler
Disallow: /
User-agent: WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Disallow: /
User-agent: WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Disallow: /
Sitemap: http://www.rawpeople.com/sitemap.xml
User-agent: *
Disallow: /*/iphone/$
Disallow: /*/iphone$
Disallow: /*/mobile/$
Disallow: /*/mobile$
Disallow: /*/mobi/$
Disallow: /*/mobi$
Disallow: /*/m/$
Disallow: /*/m$
User-agent: Aboundexbot
Disallow: /
User-agent: OutfoxBot/0.5
User-agent: complex_network_group
User-agent: imparser09.yandex.ru
User-agent: spider14.yandex.ru
User-agent: yandex.ru
User-agent: yandex
User-agent: Alexibot
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: BackDoorBot/1.0
User-agent: BPImageWalker/2.0
User-agent: Black.Hole
User-agent: BlackWidow
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: Bot mailto:craftbot@yahoo.com
User-agent: BotALot
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: DISCo
User-agent: DISCo Pump 3.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: DittoSpyder
User-agent: Download Demon
User-agent: Download Demon/3.2.0.8
User-agent: Download Demon/3.5.0.11
User-agent: EirGrabber
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Express WebPictures
User-agent: Express WebPictures (www.express-soft.com)
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: Flaming AttackBot
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: Foobot
User-agent: FrontPage
User-agent: FrontPage [NC,OR]
User-agent: Gaisbot
User-agent: GetRight
User-agent: GetRight/2.11
User-agent: GetRight/3.1
User-agent: GetRight/3.2
User-agent: GetRight/3.3
User-agent: GetRight/3.3.3
User-agent: GetRight/3.3.4
User-agent: GetRight/4.0.0
User-agent: GetRight/4.1.0
User-agent: GetRight/4.1.1
User-agent: GetRight/4.1.2
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b1
User-agent: GetRight/4.5b2
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go!Zilla 3.3 (www.gozilla.com)
User-agent: Go!Zilla 3.5 (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: HTTrack 3.0x
User-agent: HTTrack [NC,OR]
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: Indy Library
User-agent: Indy Library [NC,OR]
User-agent: InfoNaviRobot
User-agent: InterGET
User-agent: Internet Ninja
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: Iron33/1.0.2
User-agent: JOC Web Spider
User-agent: JennyBot
User-agent: JetCar
User-agent: Kenjin Spider
User-agent: Kenjin.Spider
User-agent: Keyword Density/0.9
User-agent: Keyword.Density
User-agent: LNSpiderguy
User-agent: LeechFTP
User-agent: LexiBot
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: Mister PiX
User-agent: Mister PiX version.dll
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister.PiX
User-agent: NICErsPRO
User-agent: NPBot
User-agent: NPbot
User-agent: Navroad
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.10
User-agent: NetAnts/1.23
User-agent: NetAnts/1.24
User-agent: NetAnts/1.25
User-agent: NetMechanic
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.1
User-agent: Offline Explorer/2.3
User-agent: Offline Explorer/2.4
User-agent: Offline Explorer/2.5
User-agent: Offline Navigator
User-agent: Offline.Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Oracle Ultra Search
User-agent: PageGrabber
User-agent: Papa Foto
User-agent: PerMan
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QueryN.Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: ReGet
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2.0
User-agent: WebCopier
User-agent: WebCopier v.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website eXtractor
User-agent: Website eXtractor (http:/www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: Wget
User-agent: Wget/1.10.2
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1+cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmos
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb/clsHTTP
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: tAkeOut
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
#
# ACHTUNG!!!
#
# An dieser Datei KEINE Aenderungen vornehmen!!!
# Dies ist eine automatisch erstellte Datei!
# Aenderungen werden ohne Sicherung automatisch
# ueberschrieben!
#
#-------------------------------------------------
# Version: 2013-11-13 19:35
# Generiert: 13.11.2013 19:35
# durch Script: esRemoteAccessSettings.class.php
#-------------------------------------------------
# 80legs (http://www.80legs.com/webcrawler.html)
User-agent: 008
Disallow: /
# Abonti (http://www.abonti.com)
User-agent: Abonti
Disallow: /
# AcoonBot (http://www.acoon.de/robot.asp)
User-agent: Acoon
Disallow: /
User-agent: AcoonBot
Disallow: /
User-agent: OpenAcoon
Disallow: /
# Ahrefs.com (http://ahrefs.com/robot/)
user-agent: AhrefsBot
disallow: /
# alexa.com (http://www.alexa.com/site/help/webmasters)
User-agent: ia_archiver
Disallow: /
# BacklinkCrawler (http://www.backlinktest.com/crawler.html)
User-agent: BacklinkCrawler
Disallow: /
# Baiduspider (http://www.baidu.com/search/spider.htm)
User-agent: baiduspider
Disallow: /
# Birubot [Bot]
User-agent: Birubot
Disallow: /
# Bixo Labs (http://bixolabs.com)
User-agent: bixolabs
disallow: /
# BLEXBot (http://webmeup.com/crawler.html)
User-agent: BLEXBot
Disallow: /
# BotOnParade (www.bots-on-para.de/bot.html)
User-agent: BotOnParade
Disallow: /
# BUbiNG (http://law.di.unimi.it/BUbiNG.html)
User-agent: BUbiNG
Disallow: /
# CareerBot (http://www.career-x.de/bot.html)
User-agent: careerbot
Disallow: /
# discobot (http://discoveryengine.com/discobot.html)
User-agent: discobot
Disallow: /
# Exabot/3.0 (http://www.exabot.com/go/robot)
User-agent: Exabot
Disallow: /
# Gigabot (http://www.gigablast.com/spider.html)
User-agent: gigabot
Disallow: /
# Gnip [Bot] (http://www.gnip.com)
User-agent: UnwindFetchor
Disallow: /
# Huawei Symantec (http://www.huaweisymantec.com/en/IRL/spider/)
User-agent: HuaweiSymantecSpider
Disallow: /
# iCjobs Stellenangebote Jobs (www.icjobs.de/bot.htm)
User-agent: ICCrawler - iCjobs
Disallow: /
# Linguee Bot (http://www.linguee.com/bot)
User-agent: Linguee
Disallow: /
# Majestic-12 (http://www.majestic12.co.uk/bot.php)
User-agent: MJ12bot
Disallow: /
# MojeekBot/0.2 (http://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /
# NerdByNature (http://www.nerdbynature.net/bot)
User-agent: NerdByNature.Bot
Disallow: /
# oBot/2.3.1 (http://filterdb.iss.net/crawler/)
user-agent: oBot
disallow: /
# OneRiot [Bot] (http://www.oneriot.com)
User-agent: OneRiot
disallow: /
# Plukkie (http://www.botje.com/plukkie.htm)
User-agent: plukkie
Disallow: /
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /contactus.htm
Disallow: /index.htm
Disallow: /store.htm
User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /tmp/
User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /tmp/
User-agent: Baiduspider
Disallow: /
User-agent: *
Crawl-delay: 30
# Ruby
User-agent: Ruby
disallow: /
# ScoutJet (http://www.scoutjet.com/)
User-agent: ScoutJet
Disallow: /
# Search17Bot (http://www.search17.com/bot.php)
User-agent: search17
Disallow: /
# SearchmetricsBot (http://www.searchmetrics.com/en/searchmetrics-bot/)
User-agent: SearchmetricsBot
Disallow: /
# SEOkicks-Robot (http://www.seokicks.de/robot.html)
User-agent: seokicks
Disallow: /
# seoprofiler (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Sosospider (http://help.soso.com/webspider.htm)
User-agent: Sosospider
Disallow: /
# spbot/seoprofiler.com (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
User-agent: Speedy
Disallow: /
# Spinn3r (http://spinn3r.com/robot)
User-agent: Spinn3r
Disallow: /
# The ShopWiki Crawler (http://www.shopwiki.com/wiki/Help:Bot)
User-agent: ShopWikiDisallow: /
# TurnitinBot (http://www.turnitin.com/robot/crawlerinfo.html)
User-agent: TurnitinBot
Disallow: /
# TwengaBot (http://www.twenga.com/bot.html)
User-agent: twenga2.com
Disallow: /
User-agent: twenga.com
Disallow: /
# WBSearchBot (http://www.warebay.com/bot.html)
User-agent: WBSearchBot
Disallow: /
# Yahoo! Slurp (http://help.yahoo.com/help/us/ysearch/slurp)
User-agent: Yahoo! Slurp
Disallow: /
# Yandex (http://yandex.com/bots)
User-agent: Yandex
Disallow: /
# Zookabot (http://zookabot.com)
User-agent: Zookabot
Disallow: /
#
# Erweiterungen des Shopbetreibers
#
# robots.txt fuer http://www.sofaonline24.de
#Diese Bots wollen wir sperren
User-agent: gigabot
User-agent: psbot
User-agent: Fasterfox
User-agent: vscooter
User-agent: Teleport*
User-agent: Webwhacker*
User-agent: Webzip*
User-agent: Net Attache*
User-agent: SiteSnagger*
User-agent: HTTrack*
User-agent: WebCapture*
User-agent: WebSauger*
Disallow: /
#google darf viel aber nicht alles
User-agent: Googlebot
Disallow: /export/
Disallow: /*news_detail_print
Disallow: /*order_select
Disallow: /*order_shippay
Disallow: /*order_input
Disallow: /*order_check
Disallow: /*order_ok
Disallow: /*order_wrong
Disallow: /*order_save
Disallow: /*change
Disallow: /*favorites
Disallow: /*basket_add
Disallow: /*basket_add_hidden
Disallow: /*basket_show
Disallow: /*basket_remark
Disallow: /*basket_remark_save
Disallow: /*basket_delete
Disallow: /*basket_change_quantity
Disallow: /*logout
Disallow: /*.pdf$
#inktomi ist zu gierig und darf nicht zu oft vorbeikommen
User-agent: Slurp
Crawl-Delay: 36000
Disallow: /images/
Disallow: /export/
User-agent: bingbot
Crawl-delay: 30
#hier sollten die Seiten und Verzeichnisse stehen an die keiner darf
User-agent: *
Disallow: /export/
User-agent: *
Disallow: /report/
Disallow: /reports/
Disallow: /foren/admin/
Disallow: /foren/attach_mod/
Disallow: /foren/db/
Disallow: /foren/files/
Disallow: /foren/includes/
Disallow: /foren/language/
Disallow: /foren/spelling/
Disallow: /foren/common.php
Disallow: /foren/config.php
Disallow: /foren/groupcp.php
Disallow: /foren/memberlist/
Disallow: /foren/memberlist.php
Disallow: /foren/memberlist.html
Disallow: /foren/mini_cal.php
Disallow: /foren/modcp/
Disallow: /foren/posting/
Disallow: /foren/posting.php
Disallow: /foren/posting/mode-reply/
Disallow: /foren/posting/mode-quote/
Disallow: /foren/privmsg/
Disallow: /foren/profile/
Disallow: /foren/profile.php
Disallow: /foren/ranks.php
Disallow: /foren/search/
Disallow: /foren/search.php
Disallow: /foren/statistics.php
Disallow: /foren/viewonline.php
Disallow: /foren/login.php
Disallow: /foren/login/
Disallow: /foren/*?
Disallow: /outbound.pl
Disallow: /foren_BISHER_2_0_1/
Disallow: /foren2/
Disallow: /foren/*?printertopic=
Disallow: /?*
Disallow: /seoblog/*login.php
Disallow: /seoblog/*/comment-page*
Disallow: /cgi-bin/User-agent: YandeG 1.02
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
#
# ACHTUNG!!!
#
# An dieser Datei KEINE Aenderungen vornehmen!!!
# Dies ist eine automatisch erstellte Datei!
# Aenderungen werden ohne Sicherung automatisch
# ueberschrieben!
#
#-------------------------------------------------
# Version: 2013-11-13 19:35
# Generiert: 13.11.2013 19:35
# durch Script: esRemoteAccessSettings.class.php
#-------------------------------------------------
# 80legs (http://www.80legs.com/webcrawler.html)
User-agent: 008
Disallow: /
# Abonti (http://www.abonti.com)
User-agent: Abonti
Disallow: /
# AcoonBot (http://www.acoon.de/robot.asp)
User-agent: Acoon
Disallow: /
User-agent: AcoonBot
Disallow: /
User-agent: OpenAcoon
Disallow: /
# Ahrefs.com (http://ahrefs.com/robot/)
user-agent: AhrefsBot
disallow: /
# alexa.com (http://www.alexa.com/site/help/webmasters)
User-agent: ia_archiver
Disallow: /
# BacklinkCrawler (http://www.backlinktest.com/crawler.html)
User-agent: BacklinkCrawler
Disallow: /
# Baiduspider (http://www.baidu.com/search/spider.htm)
User-agent: baiduspider
Disallow: /
# Birubot [Bot]
User-agent: Birubot
Disallow: /
# Bixo Labs (http://bixolabs.com)
User-agent: bixolabs
disallow: /
# BLEXBot (http://webmeup.com/crawler.html)
User-agent: BLEXBot
Disallow: /
# BotOnParade (www.bots-on-para.de/bot.html)
User-agent: BotOnParade
Disallow: /
# BUbiNG (http://law.di.unimi.it/BUbiNG.html)
User-agent: BUbiNG
Disallow: /
# CareerBot (http://www.career-x.de/bot.html)
User-agent: careerbot
Disallow: /
# discobot (http://discoveryengine.com/discobot.html)
User-agent: discobot
Disallow: /
# Exabot/3.0 (http://www.exabot.com/go/robot)
User-agent: Exabot
Disallow: /
# Gigabot (http://www.gigablast.com/spider.html)
User-agent: gigabot
Disallow: /
# Gnip [Bot] (http://www.gnip.com)
User-agent: UnwindFetchor
Disallow: /
# Huawei Symantec (http://www.huaweisymantec.com/en/IRL/spider/)
User-agent: HuaweiSymantecSpider
Disallow: /
# iCjobs Stellenangebote Jobs (www.icjobs.de/bot.htm)
User-agent: ICCrawler - iCjobs
Disallow: /
# Linguee Bot (http://www.linguee.com/bot)
User-agent: Linguee
Disallow: /
# Majestic-12 (http://www.majestic12.co.uk/bot.php)
User-agent: MJ12bot
Disallow: /
# MojeekBot/0.2 (http://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /
# NerdByNature (http://www.nerdbynature.net/bot)
User-agent: NerdByNature.Bot
Disallow: /
# oBot/2.3.1 (http://filterdb.iss.net/crawler/)
user-agent: oBot
disallow: /
# OneRiot [Bot] (http://www.oneriot.com)
User-agent: OneRiot
disallow: /
# Plukkie (http://www.botje.com/plukkie.htm)
User-agent: plukkie
Disallow: /
# Ruby
User-agent: Ruby
disallow: /
# ScoutJet (http://www.scoutjet.com/)
User-agent: ScoutJet
Disallow: /
# Search17Bot (http://www.search17.com/bot.php)
User-agent: search17
Disallow: /
# SearchmetricsBot (http://www.searchmetrics.com/en/searchmetrics-bot/)
User-agent: SearchmetricsBot
Disallow: /
# SEOkicks-Robot (http://www.seokicks.de/robot.html)
User-agent: seokicks
Disallow: /
# seoprofiler (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Sosospider (http://help.soso.com/webspider.htm)
User-agent: Sosospider
Disallow: /
User-agent: OutfoxBot/0.5
User-agent: complex_network_group
User-agent: imparser09.yandex.ru
User-agent: spider14.yandex.ru
User-agent: yandex.ru
User-agent: yandex
User-agent: Alexibot
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: BackDoorBot/1.0
User-agent: BPImageWalker/2.0
User-agent: Black.Hole
User-agent: BlackWidow
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: Bot mailto:craftbot@yahoo.com
User-agent: BotALot
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: DISCo
User-agent: DISCo Pump 3.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: DittoSpyder
User-agent: Download Demon
User-agent: Download Demon/3.2.0.8
User-agent: Download Demon/3.5.0.11
User-agent: EirGrabber
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Express WebPictures
User-agent: Express WebPictures (www.express-soft.com)
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: Flaming AttackBot
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: Foobot
User-agent: FrontPage
User-agent: FrontPage [NC,OR]
User-agent: Gaisbot
User-agent: GetRight
User-agent: GetRight/2.11
User-agent: GetRight/3.1
User-agent: GetRight/3.2
User-agent: GetRight/3.3
User-agent: GetRight/3.3.3
User-agent: GetRight/3.3.4
User-agent: GetRight/4.0.0
User-agent: GetRight/4.1.0
User-agent: GetRight/4.1.1
User-agent: GetRight/4.1.2
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b1
User-agent: GetRight/4.5b2
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go!Zilla 3.3 (www.gozilla.com)
User-agent: Go!Zilla 3.5 (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: HTTrack 3.0x
User-agent: HTTrack [NC,OR]
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: Indy Library
User-agent: Indy Library [NC,OR]
User-agent: InfoNaviRobot
User-agent: InterGET
User-agent: Internet Ninja
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: Iron33/1.0.2
User-agent: JOC Web Spider
User-agent: JennyBot
User-agent: JetCar
User-agent: Kenjin Spider
User-agent: Kenjin.Spider
User-agent: Keyword Density/0.9
User-agent: Keyword.Density
User-agent: LNSpiderguy
User-agent: LeechFTP
User-agent: LexiBot
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: Mister PiX
User-agent: Mister PiX version.dll
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister.PiX
User-agent: NICErsPRO
User-agent: NPBot
User-agent: NPbot
User-agent: Navroad
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.10
User-agent: NetAnts/1.23
User-agent: NetAnts/1.24
User-agent: NetAnts/1.25
User-agent: NetMechanic
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.1
User-agent: Offline Explorer/2.3
User-agent: Offline Explorer/2.4
User-agent: Offline Explorer/2.5
User-agent: Offline Navigator
User-agent: Offline.Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Oracle Ultra Search
User-agent: PageGrabber
User-agent: Papa Foto
User-agent: PerMan
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QueryN.Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: ReGet
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2.0
User-agent: WebCopier
User-agent: WebCopier v.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website eXtractor
User-agent: Website eXtractor (http:/www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: Wget
User-agent: Wget/1.10.2
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1+cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmos
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb/clsHTTP
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: tAkeOut
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
# spbot/seoprofiler.com (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
User-agent: Speedy
Disallow: /
# Spinn3r (http://spinn3r.com/robot)
User-agent: Spinn3r
Disallow: /
# The ShopWiki Crawler (http://www.shopwiki.com/wiki/Help:Bot)
User-agent: ShopWikiDisallow: /
# TurnitinBot (http://www.turnitin.com/robot/crawlerinfo.html)
User-agent: TurnitinBot
Disallow: /
# TwengaBot (http://www.twenga.com/bot.html)
User-agent: twenga2.com
Disallow: /
User-agent: twenga.com
Disallow: /
# WBSearchBot (http://www.warebay.com/bot.html)
User-agent: WBSearchBot
Disallow: /
# Yahoo! Slurp (http://help.yahoo.com/help/us/ysearch/slurp)
User-agent: Yahoo! Slurp
Disallow: /
# Yandex (http://yandex.com/bots)
User-agent: Yandex
Disallow: /
# Zookabot (http://zookabot.com)
User-agent: Zookabot
Disallow: /
#
# Erweiterungen des Shopbetreibers
#
# robots.txt fuer http://www.sofaonline24.de
#Diese Bots wollen wir sperren
User-agent: gigabot
User-agent: psbot
User-agent: Fasterfox
User-agent: vscooter
User-agent: Teleport*
User-agent: Webwhacker*
User-agent: Webzip*
User-agent: Net Attache*
User-agent: SiteSnagger*
User-agent: HTTrack*
User-agent: WebCapture*
User-agent: WebSauger*
Disallow: /
#google darf viel aber nicht alles
User-agent: Googlebot
Disallow: /export/
Disallow: /*news_detail_print
Disallow: /*order_select
Disallow: /*order_shippay
Disallow: /*order_input
Disallow: /*order_check
Disallow: /*order_ok
Disallow: /*order_wrong
Disallow: /*order_save
Disallow: /*change
Disallow: /*favorites
Disallow: /*basket_add
Disallow: /*basket_add_hidden
Disallow: /*basket_show
Disallow: /*basket_remark
Disallow: /*basket_remark_save
Disallow: /*basket_delete
Disallow: /*basket_change_quantity
Disallow: /*logout
Disallow: /*.pdf$
#inktomi ist zu gierig und darf nicht zu oft vorbeikommen
User-agent: Slurp
Crawl-Delay: 36000
Disallow: /images/
Disallow: /export/
User-agent: bingbot
Crawl-delay: 30
#hier sollten die Seiten und Verzeichnisse stehen an die keiner darf
User-agent: *
Disallow: /export/
User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout
# oder
User-agent: *
Disallow: /search
Crawl-delay: 4.5 # specifies a 4.5 second timeout
User-agent: TwengaBot
Disallow: /
# gro�e russische Suchmaschine
User-agent: Yandex
Disallow: /
# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
http://stop-spamers-and-hackers.blogspot.ch/
User-agent: *
Disallow
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml
# Ganze Site f�r alle Robots sperren
User-agent: *
Disallow: /
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
# robots.txt
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control - 5.01.4511
Disallow:/
User-agent: Microsoft URL Control - 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/
# Robots.txt file for http://www.jaguar-zelja.com
#
# Restrictions to all bots
# ------------------------------------------------
User-agent: *
Disallow: /adredir.php?
Disallow: /biz_redir.php?
Disallow: /redirect.php?
Disallow: /admin/
Disallow: /mm-data/
Disallow: /js/
Disallow: /css/
Disallow: /img/
Disallow: /join/
Disallow: /signin?
Disallow: /timthumb.php?
Disallow: /messages?
Disallow: /restaurant/join.php?
Disallow: /restaurant/signup
Disallow: /users/flag_content.php?
Disallow: /users/biz_sendToFriend.php?
Disallow: /faq.php
Disallow: /restaurant/manage_reviews.php
Disallow: /writeareview/restaurants
#
# Image Bot Blocks
# ------------------------------------------------
User-agent: googlebot-image
User-agent: psbot
User-agent: Yahoo-MMCrawler
Disallow: /
#
# Known Bots that stress the site
# ------------------------------------------------
# 070130 - Majestic12 Distributed Search Engine denied due to load/security
User-Agent: MJ12bot
Disallow: /
User-agent: ia_archiver
User-agent: ia_archiver*
User-agent: Linknzbot 2004
User-agent: Linknzbot*
User-agent: Linknzbot
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft URL Control - 6.01.9782
User-agent: Microsoft URL Control*
User-agent: Microsoft URL Control
User-agent: aipbot/1.0
User-agent: aipbot*
User-agent: aipbot
User-agent: LWP*
User-agent: LWP
User-agent: NutchCVS/0.06-dev
User-agent: Nutch*
User-agent: Nutch
Disallow: /
#
# General Bad Bots or Site scrapers that are completely disallowed
# ------------------------------------------------
User-agent: Alexibot
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: Black Hole
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: CherryPickerElite/1.0
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: Download Demon/3.2.0.8
User-agent: dumbot
User-agent: eCatch/3.0
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: FrontPage [NC,OR]
User-agent: GetRight/3.1
User-agent: GetRight/3.3
User-agent: GetRight/4.1.0
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: grub
User-agent: grub-client
User-agent: Harvest
User-agent: HMView
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack [NC,OR]
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: InfoNaviRobot
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: JetCar
User-agent: JOC Web Spider
User-agent: Kenjin Spider
User-agent: Keyword.Density
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: libWeb/clsHTTPDisallow: /
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a Unix Disallow: /
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LNSpiderguy
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister PiX version.dll
User-agent: Mister.PiX
User-agent: moget
User-agent: moget/2.1
User-agent: MSIECrawler
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.25
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http://www.innerprise.net/usp-spider.asp)
User-agent: NICErsPRO
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.4
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gathere
User-agent: pavuk
User-agent: pcBrowser
User-agent: Python-urllib
User-agent: Radiation Retriever 1.1
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: ReGet
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RMA
User-agent: searchpreview
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: sootle
User-agent: SpankBot
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: toCrawl/UrlDispatcher
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URLy Warning
User-agent: VoidEYE
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: WebCopier v2.5
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebmasterWorldForumBot
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: WebStripper/2.13
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebZIP/2.75 (http://www.spidersoft.com)
User-agent: WebZIP/3.65 (http://www.spidersoft.com)
User-agent: WebZIP/3.80 (http://www.spidersoft.com)
User-agent: WebZip/4.0
User-agent: WebZIP/4.0 (http://www.spidersoft.com)
User-agent: WebZIP/4.1 (http://www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http://www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http://www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http://www.spidersoft.com)
User-agent: Wget
User-agent: Wget/1.5.2
User-agent: Wget/1.7
User-agent: Wget/1.8.2
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: 008
User-agent: HttpTool*
User-agent: RealWebsiteWorth
User-agent: http://www.almaden.ibm.com/cs/crawler
User-agent: Bilbo/2.3b-UNIX
User-agent: FAST-WebCrawler/3.6/FirstPage
User-agent: Missigua Locator 1.9
User-agent: Yandex
User-Agent: OmniExplorer_Bot
User-Agent: MJ12bot
User-agent: *libwww*
User-agent: *Fetch*
User-agent: BorderManager*
User-agent: Maxthon
User-agent: Java*
User-agent: msrbot
User-agent: BSpider
User-agent: legs
User-agent: exabot
Disallow: /
# Alle Robots ausschlie�en
User-agent: *
Disallow: /temp/
User-agent: Baiduspider
Disallow: /my-page.html
User-Agent: Googlebot-Mobile
Disallow: /
User-agent: Googlebot
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: Googlebot-Mobile
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: Feedfetcher-Google
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: bingbot
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
User-agent: Yahoo!
Disallow: /status/401.php
Disallow: /status/403.php
Disallow: /status/404.php
Disallow: /status/410.php
# alle nicht genannten Bots sperren
User-agent: *
Disallow: /
### Automatically banned scanners and crawlers section ###
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
### Section end ###
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
#Baiduspider
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider+
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
Disallow: /
User-agent: Baiduspider/2.0;+http://www.baidu.com/search/spider.html
Disallow: /
User-agent: Baiduspider/2.0
Disallow: /
User-agent: +Baiduspider
Disallow: /
User-agent: +Baiduspider/2.0
Disallow: /
User-agent: +Baiduspider/2.0;++http://www.baidu.com/search/spider.html
Disallow: /
User-agent: Mozilla/5.0(compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Disallow: /
#Yandex
User-agent: spider
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
#Sosospider
User-agent: Sosospider+
Disallow: /
#Ezooms
User-agent: ezooms
Disallow: /
User-agent: Ezooms/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot[at]gmail[dot]com)
Disallow: /
#Sogou
User-agent: sogou spider
Disallow: /
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
# robots.txt
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control - 5.01.4511
Disallow:/
User-agent: Microsoft URL Control - 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /404_files/
Disallow: /logs/
Disallow: /plesk-stat/
Disallow: /test/
User-Agent: Googlebot-Mobile
Disallow: /
User-agent: baiduSpider
Disallow: /
User-agent: BaiduSpider+
Disallow: /
User-agent: BaiduSpider+ (+; http://www.baidu.com/search/spider.htm; Ubuntu; Compatible; Version; Platform; .NET CLR 2.0.50727; Version)
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /
User-agent: Fatbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml)
Disallow: /
User-agent: genieBot
Disallow: /
User-agent: genieBot (http://64.5.245.11/faq/faq.html)
Disallow: /
User-agent: genieBot enash@genieknows.com
Disallow: /
User-agent: geniebot wgao@genieknows.com
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Gigabot/1.0
Disallow: /
User-agent: Gigabot/2.0
Disallow: /
User-agent: Gigabot/2.0 (http://www.gigablast.com/spider.html)
Disallow: /
User-agent: Gigabot/2.0/gigablast.com/spider.html
Disallow: /
User-agent: Gigabot/2.0; http://www.gigablast.com/spider.html
Disallow: /
User-agent: Gigabot/2.0att
Disallow: /
User-agent: Gigabot/3.0 (http://www.gigablast.com/spider.html)
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MJ12bot/v1.0.0 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.1 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.2 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.3 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.4 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.5 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.6 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.1.0 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: OmniExplorer_Bot
Disallow: /
User-agent: OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer
Disallow: /
User-agent: OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com)
Disallow: /
User-agent: OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Cars Crawler
Disallow: /
User-agent: OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Internet Indexer
Disallow: /
User-agent: OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com)
Disallow: /
User-agent: OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com) Jobs Crawler
Disallow: /
User-agent: OmniExplorer_Bot/2.47 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/2.93 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.06c (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.06d (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.10 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.10a (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.11a (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.11c (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.20b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.21 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.23 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.24 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.26b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.26c (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.28 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.94b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.95 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.98b (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/3.99 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/5.25 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/5.73 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/5.77e (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.0 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.13 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.7 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.10.9 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: OmniExplorer_Bot/6.70 (+http://www.omni-explorer.com) WorldIndexer
Disallow: /
User-agent: SBIder
Disallow: /
User-agent: SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
Disallow: /
User-agent: SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
Disallow: /
User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Disallow: /
User-agent: WebAlta Crawler
Disallow: /
User-agent: WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Disallow: /
User-agent: WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Disallow: /
Sitemap: http://www.rawpeople.com/sitemap.xml
User-agent: *
Disallow: /*/iphone/$
Disallow: /*/iphone$
Disallow: /*/mobile/$
Disallow: /*/mobile$
Disallow: /*/mobi/$
Disallow: /*/mobi$
Disallow: /*/m/$
Disallow: /*/m$
User-agent: Aboundexbot
Disallow: /
User-agent: OutfoxBot/0.5
User-agent: complex_network_group
User-agent: imparser09.yandex.ru
User-agent: spider14.yandex.ru
User-agent: yandex.ru
User-agent: yandex
User-agent: Alexibot
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: BackDoorBot/1.0
User-agent: BPImageWalker/2.0
User-agent: Black.Hole
User-agent: BlackWidow
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: Bot mailto:craftbot@yahoo.com
User-agent: BotALot
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: DISCo
User-agent: DISCo Pump 3.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: DittoSpyder
User-agent: Download Demon
User-agent: Download Demon/3.2.0.8
User-agent: Download Demon/3.5.0.11
User-agent: EirGrabber
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Express WebPictures
User-agent: Express WebPictures (www.express-soft.com)
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: Flaming AttackBot
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: Foobot
User-agent: FrontPage
User-agent: FrontPage [NC,OR]
User-agent: Gaisbot
User-agent: GetRight
User-agent: GetRight/2.11
User-agent: GetRight/3.1
User-agent: GetRight/3.2
User-agent: GetRight/3.3
User-agent: GetRight/3.3.3
User-agent: GetRight/3.3.4
User-agent: GetRight/4.0.0
User-agent: GetRight/4.1.0
User-agent: GetRight/4.1.1
User-agent: GetRight/4.1.2
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b1
User-agent: GetRight/4.5b2
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go!Zilla 3.3 (www.gozilla.com)
User-agent: Go!Zilla 3.5 (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: HTTrack 3.0x
User-agent: HTTrack [NC,OR]
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: Indy Library
User-agent: Indy Library [NC,OR]
User-agent: InfoNaviRobot
User-agent: InterGET
User-agent: Internet Ninja
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: Iron33/1.0.2
User-agent: JOC Web Spider
User-agent: JennyBot
User-agent: JetCar
User-agent: Kenjin Spider
User-agent: Kenjin.Spider
User-agent: Keyword Density/0.9
User-agent: Keyword.Density
User-agent: LNSpiderguy
User-agent: LeechFTP
User-agent: LexiBot
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: Mister PiX
User-agent: Mister PiX version.dll
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister.PiX
User-agent: NICErsPRO
User-agent: NPBot
User-agent: NPbot
User-agent: Navroad
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.10
User-agent: NetAnts/1.23
User-agent: NetAnts/1.24
User-agent: NetAnts/1.25
User-agent: NetMechanic
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.1
User-agent: Offline Explorer/2.3
User-agent: Offline Explorer/2.4
User-agent: Offline Explorer/2.5
User-agent: Offline Navigator
User-agent: Offline.Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Oracle Ultra Search
User-agent: PageGrabber
User-agent: Papa Foto
User-agent: PerMan
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QueryN.Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: ReGet
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2.0
User-agent: WebCopier
User-agent: WebCopier v.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website eXtractor
User-agent: Website eXtractor (http:/www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: Wget
User-agent: Wget/1.10.2
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1+cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmos
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb/clsHTTP
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: tAkeOut
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
#
# ACHTUNG!!!
#
# An dieser Datei KEINE Aenderungen vornehmen!!!
# Dies ist eine automatisch erstellte Datei!
# Aenderungen werden ohne Sicherung automatisch
# ueberschrieben!
#
#-------------------------------------------------
# Version: 2013-11-13 19:35
# Generiert: 13.11.2013 19:35
# durch Script: esRemoteAccessSettings.class.php
#-------------------------------------------------
# 80legs (http://www.80legs.com/webcrawler.html)
User-agent: 008
Disallow: /
# Abonti (http://www.abonti.com)
User-agent: Abonti
Disallow: /
# AcoonBot (http://www.acoon.de/robot.asp)
User-agent: Acoon
Disallow: /
User-agent: AcoonBot
Disallow: /
User-agent: OpenAcoon
Disallow: /
# Ahrefs.com (http://ahrefs.com/robot/)
user-agent: AhrefsBot
disallow: /
# alexa.com (http://www.alexa.com/site/help/webmasters)
User-agent: ia_archiver
Disallow: /
# BacklinkCrawler (http://www.backlinktest.com/crawler.html)
User-agent: BacklinkCrawler
Disallow: /
# Baiduspider (http://www.baidu.com/search/spider.htm)
User-agent: baiduspider
Disallow: /
# Birubot [Bot]
User-agent: Birubot
Disallow: /
# Bixo Labs (http://bixolabs.com)
User-agent: bixolabs
disallow: /
# BLEXBot (http://webmeup.com/crawler.html)
User-agent: BLEXBot
Disallow: /
# BotOnParade (www.bots-on-para.de/bot.html)
User-agent: BotOnParade
Disallow: /
# BUbiNG (http://law.di.unimi.it/BUbiNG.html)
User-agent: BUbiNG
Disallow: /
# CareerBot (http://www.career-x.de/bot.html)
User-agent: careerbot
Disallow: /
# discobot (http://discoveryengine.com/discobot.html)
User-agent: discobot
Disallow: /
# Exabot/3.0 (http://www.exabot.com/go/robot)
User-agent: Exabot
Disallow: /
# Gigabot (http://www.gigablast.com/spider.html)
User-agent: gigabot
Disallow: /
# Gnip [Bot] (http://www.gnip.com)
User-agent: UnwindFetchor
Disallow: /
# Huawei Symantec (http://www.huaweisymantec.com/en/IRL/spider/)
User-agent: HuaweiSymantecSpider
Disallow: /
# iCjobs Stellenangebote Jobs (www.icjobs.de/bot.htm)
User-agent: ICCrawler - iCjobs
Disallow: /
# Linguee Bot (http://www.linguee.com/bot)
User-agent: Linguee
Disallow: /
# Majestic-12 (http://www.majestic12.co.uk/bot.php)
User-agent: MJ12bot
Disallow: /
# MojeekBot/0.2 (http://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /
# NerdByNature (http://www.nerdbynature.net/bot)
User-agent: NerdByNature.Bot
Disallow: /
# oBot/2.3.1 (http://filterdb.iss.net/crawler/)
user-agent: oBot
disallow: /
# OneRiot [Bot] (http://www.oneriot.com)
User-agent: OneRiot
disallow: /
# Plukkie (http://www.botje.com/plukkie.htm)
User-agent: plukkie
Disallow: /
# Ruby
User-agent: Ruby
disallow: /
# ScoutJet (http://www.scoutjet.com/)
User-agent: ScoutJet
Disallow: /
# Search17Bot (http://www.search17.com/bot.php)
User-agent: search17
Disallow: /
# SearchmetricsBot (http://www.searchmetrics.com/en/searchmetrics-bot/)
User-agent: SearchmetricsBot
Disallow: /
# SEOkicks-Robot (http://www.seokicks.de/robot.html)
User-agent: seokicks
Disallow: /
# seoprofiler (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Sosospider (http://help.soso.com/webspider.htm)
User-agent: Sosospider
Disallow: /
# spbot/seoprofiler.com (http://www.seoprofiler.com/bot)
User-agent: spbot
Disallow: /
# Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
User-agent: Speedy
Disallow: /
# Spinn3r (http://spinn3r.com/robot)
User-agent: Spinn3r
Disallow: /
# The ShopWiki Crawler (http://www.shopwiki.com/wiki/Help:Bot)
User-agent: ShopWikiDisallow: /
# TurnitinBot (http://www.turnitin.com/robot/crawlerinfo.html)
User-agent: TurnitinBot
Disallow: /
User-agent:Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Disalow:/
# TwengaBot (http://www.twenga.com/bot.html)
User-agent: twenga2.com
Disallow: /
User-agent: twenga.com
Disallow: /
# WBSearchBot (http://www.warebay.com/bot.html)
User-agent: WBSearchBot
Disallow: /
# Yahoo! Slurp (http://help.yahoo.com/help/us/ysearch/slurp)
User-agent: Yahoo! Slurp
Disallow: /
# Yandex (http://yandex.com/bots)
User-agent: Yandex
Disallow: /
# Zookabot (http://zookabot.com)
User-agent: Zookabot
Disallow: /
#
# Erweiterungen des Shopbetreibers
#
# robots.txt fuer http://www.sofaonline24.de
#Diese Bots wollen wir sperren
User-agent: gigabot
User-agent: psbot
User-agent: Fasterfox
User-agent: vscooter
User-agent: Teleport*
User-agent: Webwhacker*
User-agent: Webzip*
User-agent: Net Attache*
User-agent: SiteSnagger*
User-agent: HTTrack*
User-agent: WebCapture*
User-agent: WebSauger*
Disallow: /
#google darf viel aber nicht alles
User-agent: Googlebot
Disallow: /export/
Disallow: /*news_detail_print
Disallow: /*order_select
Disallow: /*order_shippay
Disallow: /*order_input
Disallow: /*order_check
Disallow: /*order_ok
Disallow: /*order_wrong
Disallow: /*order_save
Disallow: /*change
Disallow: /*favorites
Disallow: /*basket_add
Disallow: /*basket_add_hidden
Disallow: /*basket_show
Disallow: /*basket_remark
Disallow: /*basket_remark_save
Disallow: /*basket_delete
Disallow: /*basket_change_quantity
Disallow: /*logout
Disallow: /*.pdf$
#inktomi ist zu gierig und darf nicht zu oft vorbeikommen
User-agent: Slurp
Crawl-Delay: 36000
Disallow: /images/
Disallow: /export/
User-agent: bingbot
Crawl-delay: 30
#hier sollten die Seiten und Verzeichnisse stehen an die keiner darf
User-agent: *
Disallow: /export/
User-agent: *
Disallow: /report/
Disallow: /reports/
Disallow: /foren/admin/
Disallow: /foren/attach_mod/
Disallow: /foren/db/
Disallow: /foren/files/
Disallow: /foren/includes/
Disallow: /foren/language/
Disallow: /foren/spelling/
Disallow: /foren/common.php
Disallow: /foren/config.php
Disallow: /foren/groupcp.php
Disallow: /foren/memberlist/
Disallow: /foren/memberlist.php
Disallow: /foren/memberlist.html
Disallow: /foren/mini_cal.php
Disallow: /foren/modcp/
Disallow: /foren/posting/
Disallow: /foren/posting.php
Disallow: /foren/posting/mode-reply/
Disallow: /foren/posting/mode-quote/
Disallow: /foren/privmsg/
Disallow: /foren/profile/
Disallow: /foren/profile.php
Disallow: /foren/ranks.php
Disallow: /foren/search/
Disallow: /foren/search.php
Disallow: /foren/statistics.php
Disallow: /foren/viewonline.php
Disallow: /foren/login.php
Disallow: /foren/login/
Disallow: /foren/*?
Disallow: /outbound.pl
Disallow: /foren_BISHER_2_0_1/
Disallow: /foren2/
Disallow: /foren/*?printertopic=
Disallow: /?*
Disallow: /seoblog/*login.php
Disallow: /seoblog/*/comment-page*
Disallow: /cgi-bin/
User-agent: Googlebot
Disallow: /
User-agent: Mediapartners-Google
Disallow:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-Image
Disallow: /personal
User-agent: Yandex
Disallow: / #
User-agent: *
Disallow: /
User-agent: Googlebot
Disallow:
User-agent: Yahoo-slurp
Disallow:
User-agent: Msnbot
Disallow:
User-agent: *
Disallow: /adredir.php?
Disallow: /biz_redir.php?
Disallow: /redirect.php?
Disallow: /admin/
Disallow: /mm-data/
Disallow: /js/
Disallow: /css/
Disallow: /img/
Disallow: /join/
Disallow: /signin?
Disallow: /timthumb.php?
Disallow: /messages?
Disallow: /restaurant/join.php?
Disallow: /restaurant/signup
Disallow: /users/flag_content.php?
Disallow: /users/biz_sendToFriend.php?
Disallow: /faq.php
Disallow: /restaurant/manage_reviews.php
Disallow: /writeareview/restaurants
# Robots.txt file for http://www.jaguar-zelja.com
#
# Restrictions to all bots
# ------------------------------------------------
#
# Image Bot Blocks
# ------------------------------------------------
User-agent: googlebot-image
User-agent: psbot
User-agent: Yahoo-MMCrawler
Disallow: /
#
# Known Bots that stress the site
# ------------------------------------------------
# 070130 - Majestic12 Distributed Search Engine denied due to load/security
User-Agent: MJ12bot
Disallow: /
User-agent: ia_archiver
User-agent: ia_archiver*
User-agent: Linknzbot 2004
User-agent: Linknzbot*
User-agent: Linknzbot
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft URL Control - 6.01.9782
User-agent: Microsoft URL Control*
User-agent: Microsoft URL Control
User-agent: aipbot/1.0
User-agent: aipbot*
User-agent: aipbot
User-agent: LWP*
User-agent: LWP
User-agent: NutchCVS/0.06-dev
User-agent: Nutch*
User-agent: Nutch
Disallow: /
#
# General Bad Bots or Site scrapers that are completely disallowed
# ------------------------------------------------
User-agent: Alexibot
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: Black Hole
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: CherryPickerElite/1.0
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: Download Demon/3.2.0.8
User-agent: dumbot
User-agent: eCatch/3.0
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: FrontPage [NC,OR]
User-agent: GetRight/3.1
User-agent: GetRight/3.3
User-agent: GetRight/4.1.0
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: grub
User-agent: grub-client
User-agent: Harvest
User-agent: HMView
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack [NC,OR]
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: InfoNaviRobot
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: JetCar
User-agent: JOC Web Spider
User-agent: Kenjin Spider
User-agent: Keyword.Density
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: libWeb/clsHTTPDisallow: /
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a Unix Disallow: /
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LNSpiderguy
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister PiX version.dll
User-agent: Mister.PiX
User-agent: moget
User-agent: moget/2.1
User-agent: MSIECrawler
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.25
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http://www.innerprise.net/usp-spider.asp)
User-agent: NICErsPRO
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.4
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gathere
User-agent: pavuk
User-agent: pcBrowser
User-agent: Python-urllib
User-agent: Radiation Retriever 1.1
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: ReGet
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RMA
User-agent: searchpreview
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: sootle
User-agent: SpankBot
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: toCrawl/UrlDispatcher
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URLy Warning
User-agent: VoidEYE
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: WebCopier v2.5
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebmasterWorldForumBot
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: WebStripper/2.13
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebZIP/2.75 (http://www.spidersoft.com)
User-agent: WebZIP/3.65 (http://www.spidersoft.com)
User-agent: WebZIP/3.80 (http://www.spidersoft.com)
User-agent: WebZip/4.0
User-agent: WebZIP/4.0 (http://www.spidersoft.com)
User-agent: WebZIP/4.1 (http://www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http://www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http://www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http://www.spidersoft.com)
User-agent: Wget
User-agent: Wget/1.5.2
User-agent: Wget/1.7
User-agent: Wget/1.8.2
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: 008
User-agent: HttpTool*
User-agent: RealWebsiteWorth
User-agent: http://www.almaden.ibm.com/cs/crawler
User-agent: Bilbo/2.3b-UNIX
User-agent: FAST-WebCrawler/3.6/FirstPage
User-agent: Missigua Locator 1.9
User-agent: Yandex
User-Agent: OmniExplorer_Bot
User-Agent: MJ12bot
User-agent: *libwww*
User-agent: *Fetch*
User-agent: BorderManager*
User-agent: Maxthon
User-agent: Java*
User-agent: msrbot
User-agent: BSpider
User-agent: legs
User-agent: exabot
Disallow: /
User-agent:Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Disalow:/
#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used: http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Crawl-delay: 30
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /
User-agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Disallow: /
User-agent: AcoonBot/4.11.1
Disallow: /
User-agent: ezooms.bot@gmail.com
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Ezooms/1.0
Disallow: /
User-agent: Ezooms/1.0; ezooms.bot@gmail.com
Disallow: /
User-agent: Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MJ12bot/v1.4.3
Disallow: /
User-agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: Mozilla/5.0+(compatible;+WebofantBot+v3.15;+http://www.webofant.ch)
Disallow: /
User-agent: VoilaBot*
Disallow: /
User-agent: Voilabot*
Disallow: /
User-agent: VoilaBot BETA 1.2*
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
Disallow: /
User-agent: x28-job-bot; +http://x28.ch/bot.html
Disallow: /
User-agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Disallow: /
#
User-agent: YandexBot
Disallow: / admin
User-agent: OutfoxBot/0.5
User-agent: complex_network_group
User-agent: imparser09.yandex.ru
User-agent: spider14.yandex.ru
User-agent: yandex.ru
User-agent: yandex
User-agent: Alexibot
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: BackDoorBot/1.0
User-agent: BPImageWalker/2.0
User-agent: Black.Hole
User-agent: BlackWidow
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: Bot mailto:craftbot@yahoo.com
User-agent: BotALot
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: DISCo
User-agent: DISCo Pump 3.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: DittoSpyder
User-agent: Download Demon
User-agent: Download Demon/3.2.0.8
User-agent: Download Demon/3.5.0.11
User-agent: EirGrabber
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Express WebPictures
User-agent: Express WebPictures (www.express-soft.com)
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: Flaming AttackBot
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: Foobot
User-agent: FrontPage
User-agent: FrontPage [NC,OR]
User-agent: Gaisbot
User-agent: GetRight
User-agent: GetRight/2.11
User-agent: GetRight/3.1
User-agent: GetRight/3.2
User-agent: GetRight/3.3
User-agent: GetRight/3.3.3
User-agent: GetRight/3.3.4
User-agent: GetRight/4.0.0
User-agent: GetRight/4.1.0
User-agent: GetRight/4.1.1
User-agent: GetRight/4.1.2
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b1
User-agent: GetRight/4.5b2
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go!Zilla 3.3 (www.gozilla.com)
User-agent: Go!Zilla 3.5 (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: HTTrack 3.0x
User-agent: HTTrack [NC,OR]
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: Indy Library
User-agent: Indy Library [NC,OR]
User-agent: InfoNaviRobot
User-agent: InterGET
User-agent: Internet Ninja
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: Iron33/1.0.2
User-agent: JOC Web Spider
User-agent: JennyBot
User-agent: JetCar
User-agent: Kenjin Spider
User-agent: Kenjin.Spider
User-agent: Keyword Density/0.9
User-agent: Keyword.Density
User-agent: LNSpiderguy
User-agent: LeechFTP
User-agent: LexiBot
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: Mister PiX
User-agent: Mister PiX version.dll
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister.PiX
User-agent: NICErsPRO
User-agent: NPBot
User-agent: NPbot
User-agent: Navroad
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.10
User-agent: NetAnts/1.23
User-agent: NetAnts/1.24
User-agent: NetAnts/1.25
User-agent: NetMechanic
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.1
User-agent: Offline Explorer/2.3
User-agent: Offline Explorer/2.4
User-agent: Offline Explorer/2.5
User-agent: Offline Navigator
User-agent: Offline.Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Oracle Ultra Search
User-agent: PageGrabber
User-agent: Papa Foto
User-agent: PerMan
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QueryN.Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: ReGet
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2.0
User-agent: WebCopier
User-agent: WebCopier v.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website eXtractor
User-agent: Website eXtractor (http:/www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: Wget
User-agent: Wget/1.10.2
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1+cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmos
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb/clsHTTP
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: tAkeOut
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
Crawl-delay: 30
# Block All trafic for certain user agents
User-agent: Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
User-agent: AddThis.com robot tech.support@clearspring.com
User-agent: MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
User-agent: MSRBOT # Microsoft Research Bot
User-agent: SandCrawler # Unknown Microsoft crawler
User-agent: Shim-Crawler # Japanese university research
User-agent: ScoutJet # Not relevant yet
# AhrefsBot
User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) #Link tracker
#
User-agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
User-agent: DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, crawler@dotnetdotcom.org)DotBot # Link tracker
User-agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
# Find Links
User-agent: findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.4-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.3-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.3-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.3-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.3-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.2-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.1-a1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a9 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a8 ( http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a7 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a4 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1-a3 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.06 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.0.8 (+http://wortschatz.uni-leipzig.de/findlinks/)
User-agent: findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
#
User-agent: Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.searchtechnologies.com)
User-agent: SputnikBot
User-agent: 008
User-agent: Accoona
User-agent: AddSugarSpiderBot
User-agent: AnyApexBot
User-agent: BecomeBot #Consumer shopping
User-agent: BeslistBot # Dutch consumer shopping
User-agent: BillyBobBot # Dead site
User-agent: Bimbot # Hiding identity
User-agent: Boitho # Not relevant
User-agent: btbot # Torrent search
User-agent: CatchBot # Dead Australian search
User-agent: cosmos # not relevant
User-agent: Covario
User-agent: Cerberian # not relevant
User-agent: DiamondBot
User-agent: Discobot
User-agent: EmeraldShield.com
User-agent: envolk[ITS]spider
User-agent: EsperanzaBot
User-agent: Exabot
User-agent: FAST Enterprise Crawler
User-agent: FAST-WebCrawler
User-agent: FDSE robot
User-agent: FurlBot
User-agent: FyberSpider
User-agent: g2crawler
User-agent: Gaisbot
User-agent: GalaxyBot
User-agent: genieBot
User-agent: Gigabot
User-agent: Girafabot
User-agent: HappyFunBot
User-agent: Holmes
User-agent: htdig
User-agent: iaskspider # unknown Chinese spider
User-agent: ia_archiver # Alexa crawler
User-agent: IRLbot
User-agent: IssueCrawler
User-agent: Jaxified Bot
User-agent: Jyxobot
User-agent: KoepaBot
User-agent: L.webis
User-agent: LapozzBot
User-agent: Larbin
User-agent: LDSpider
User-agent: LDSpider
User-agent: Linguee Bot
User-agent: LinkWalker
User-agent: lmspider
User-agent: lwp-trivial
User-agent: mabontland
User-agent: magpie-crawler
User-agent: MJ12bot
User-agent: mogimogi
User-agent: MojeekBot
User-agent: Moreoverbot
User-agent: Morning Paper
User-agent: MVAClient
User-agent: mxbot
User-agent: NetResearchServer
User-agent: NetSeer Crawler
User-agent: NewsGator
User-agent: NG-Search
User-agent: nicebot
User-agent: Nusearch
User-agent: NutchCVS
User-agent: Nymesis
User-agent: obot
User-agent: oegp
User-agent: omgilibot
User-agent: OmniExplorer_Bot
User-agent: OOZBOT
User-agent: Orbiter
User-agent: PageBitesHyperBot
User-agent: Peew
User-agent: polybot
User-agent: PostPost
User-agent: Psbot - Image search
User-agent: PycURL
User-agent: Qseero
User-agent: Radian6
User-agent: RAMPyBot
User-agent: RufusBot
User-agent: SBIder
User-agent: Scrubby
User-agent: SearchSight
User-agent: Seekbot
User-agent: semanticdiscovery
User-agent: Sensis Web Crawler
User-agent: SEOChat::Bot
User-agent: ShopWiki
User-agent: Shoula robot
User-agent: silk
User-agent: Sitebot
User-agent: Snappy
User-agent: Speedy Spider
User-agent: Sqworm
User-agent: suggybot
User-agent: SurveyBot
User-agent: TerrawizBot
User-agent: TheSuBot
# TinEye
User-agent: TinEye/1.1 (http://tineye.com/crawler.html)
User-agent: TinEye
#
User-agent: truwoGPS
User-agent: TurnitinBot
User-agent: TweetedTimes Bot
User-agent: TwengaBot
User-agent: updated
User-agent: Urlfilebot
User-agent: Vagabondo
User-agent: Vortex
User-agent: voyager
User-agent: VYU2
User-agent: webcollage
User-agent: Websquash.com
User-agent: wf84
User-agent: WomlpeFactory
User-agent: Xaldon_WebSpider
User-agent: yacy
User-agent: Yasaklibot
User-agent: yoogliFetchAgent
User-agent: Zao/0.1 (http://www.kototoi.org/zao/)
User-agent: Mozilla/4.0 (compatible; Zealbot 1.0)
User-agent: zspider/0.9-dev http://feedback.redkolibri.com/
# Zyborg
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
#
Disallow: /
#------------------------------------
#Block all videos for Ultraseek
User-agent: Ultraseek
Disallow: /*video/
Disallow: /en/video/
Disallow: /ja/video/
Disallow: /fr/video/
Disallow: /zh/video/
Disallow: /de/video/
Disallow: /it/video/
Disallow: /cs/video/
Disallow: /hu/video/
Disallow: /es/video/
Disallow: /ru/video/
Disallow: /pl/video/
Disallow: /pt/video/
Disallow: /ko/video/
# -----------------------------------
# Chinese Spider - Block all languages other than Chinese and English
User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
User-agent: BaiDuSpider
User-agent: Sosospider+(+http://help.soso.com/webspider.htm)
User-agent: sogou spider
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
User-agent: Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
User-agent: Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Disallow: /ja/
Disallow: /fr/
Disallow: /de/
Disallow: /it/
Disallow: /cs/
Disallow: /hu/
Disallow: /es/
Disallow: /ru/
Disallow: /pl/
Disallow: /pt/
Disallow: /ko/
Disallow: /vi/
Disallow: /th/
# Russian Spiders
# -----------------------------------
# Russian Spiders - Block all languages other than Russian and English
User-agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
User-agent: igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
User-agent: StackRambler/2.0 (MSIE incompatible
User-agent: StackRambler/2.0
Disallow: /ja/
Disallow: /fr/
Disallow: /zh/
Disallow: /de/
Disallow: /it/
Disallow: /cs/
Disallow: /hu/
Disallow: /es/
Disallow: /pl/
Disallow: /pt/
Disallow: /ko/
Disallow: /vi/
Disallow: /th/
# --------------------------------------
# Japanese Spiders - Block all languages other than Japanese and English
User-agent: Mozilla/4.0 (compatible; Arachmo)
User-agent: ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
User-agent: ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
User-agent: ichiro/2.0 (ichiro@nttr.co.jp)
User-agent: ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
Disallow: /fr/
Disallow: /zh/
Disallow: /de/
Disallow: /it/
Disallow: /cs/
Disallow: /hu/
Disallow: /es/
Disallow: /ru/
Disallow: /pl/
Disallow: /pt/
Disallow: /ko/
Disallow: /vi/
Disallow: /th/
# ----------------------------------------
# German Spiders - Block all langauges other than German and English
User-agent: ABACHOBot
User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
User-agent: iCCrawler (http://www.iccenter.net/bot.htm)# German job listing site
User-agent: SynooBot/0.7.1 (SynooBot; http://www.synoo.de/bot.html; webmaster@synoo.com)
User-agent: fastbot.de crawler 2.0 beta (http://www.fastbot.de)
Disallow: /ja/
Disallow: /fr/
Disallow: /zh/
Disallow: /it/
Disallow: /cs/
Disallow: /hu/
Disallow: /es/
Disallow: /ru/
Disallow: /pl/
Disallow: /pt/
Disallow: /ko/
Disallow: /vi/
Disallow: /th/
# ---------------------------------
# French Spiders - Block all languages other than French and English
#Block downloads for all user agents
User-agent: Pompos/1.3 http://dir.com/pompos.html
User-agent: Pompos/1.2 http://pompos.iliad.fr
User-agent: Pompos/1.1 http://pompos.iliad.fr
User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)
Disallow: /ja/
Disallow: /zh/
Disallow: /de/
Disallow: /it/
Disallow: /cs/
Disallow: /hu/
Disallow: /es/
Disallow: /ru/
Disallow: /pl/
Disallow: /pt/
Disallow: /ko/
Disallow: /vi/
Disallow: /th/
# ----------------------------------------
# Czech Langauge - Block all languages other than Czech and English
User-agent: SeznamBot/2.0 (+http://fulltext.seznam.cz/)
User-agent: SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
Disallow: /ja/
Disallow: /fr/
Disallow: /zh/
Disallow: /de/
Disallow: /it/
Disallow: /hu/
Disallow: /es/
Disallow: /ru/
Disallow: /pl/
Disallow: /pt/
Disallow: /ko/
Disallow: /vi/
Disallow: /th/
# ----------------------------------------------
# Korean Language - Block all languages other than Korean and English
# Block downloads for all user agents
User-agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
User-agent: Yeti/1.0 (+http://help.naver.com/robots/)
User-agent: Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
Disallow: /ja/
Disallow: /fr/
Disallow: /zh/
Disallow: /de/
Disallow: /it/
Disallow: /cs/
Disallow: /hu/
Disallow: /es/
Disallow: /ru/
Disallow: /pl/
Disallow: /pt/
Disallow: /vi/
Disallow: /th/
# All User Agents
# -------------------------------
User-agent: *
#
# Slow them down
Crawl-delay: 15
#
# Disallow dynamic pages
# Disallow: /*/? # google supports wildcards -- this causes problem with adwords
#
#
Disallow: /data/File/
#
#
# Disallow dyanamic applications searches
Disallow: /*/applications/? # Google supports wildcards
Disallow: /en/applications/?
Disallow: /ja/applications/?
Disallow: /fr/applications/?
Disallow: /zh/applications/?
Disallow: /de/applications/?
Disallow: /it/applications/?
Disallow: /cs/applications/?
Disallow: /hu/applications/?
Disallow: /es/applications/?
Disallow: /ru/applications/?
Disallow: /pl/applications/?
Disallow: /pt/applications/?
Disallow: /ko/applications/?
Disallow: /vi/applications/?
Disallow: /th/applications/?
#
# Block downloads for all user agents
Disallow: /*.downloads # google supports wildcards
Disallow: /logout-member/
Disallow: /en/.downloads/
Disallow: /ja/.downloads/
Disallow: /fr/.downloads/
Disallow: /zh/.downloads/
Disallow: /de/.downloads/
Disallow: /it/.downloads/
Disallow: /cs/.downloads/
Disallow: /hu/.downloads/
Disallow: /es/.downloads/
Disallow: /ru/.downloads/
Disallow: /pl/.downloads/
Disallow: /pt/.downloads/
Disallow: /ko/.downloads/
Disallow: /vi/.downloads/
Disallow: /th/.downloads/
#
# Disallow quote request page for all
Disallow: /*/quote-request # google supports wildcards
Disallow: /en/quote-request/
Disallow: /ja/quote-request/
Disallow: /fr/quote-request/
Disallow: /zh/quote-request/
Disallow: /de/quote-request/
Disallow: /it/quote-request/
Disallow: /cs/quote-request/
Disallow: /hu/quote-request/
Disallow: /es/quote-request/
Disallow: /ru/quote-request/
Disallow: /pl/quote-request/
Disallow: /pt/quote-request/
Disallow: /ko/quote-request/
Disallow: /vi/quote-request/
Disallow: /th/quote-request/
#
# Disallow Industry pages page for all
Disallow: /*/industries # google supports wildcards
Disallow: /en/industries/
Disallow: /ja/industries/
Disallow: /fr/industries/
Disallow: /zh/industries/
Disallow: /de/industries/
Disallow: /it/industries/
Disallow: /cs/industries/
Disallow: /hu/industries/
Disallow: /es/industries/
Disallow: /ru/industries/
Disallow: /pl/industries/
Disallow: /pt/industries/
Disallow: /ko/industries/
Disallow: /vi/industries/
Disallow: /th/industries/
#
# Disallow contact us page for all
Disallow: /*/contact-us # google supports wildcards
Disallow: /en/products/contact-us/
Disallow: /ja/products/contact-us/
Disallow: /fr/products/contact-us/
Disallow: /zh/products/contact-us/
Disallow: /de/products/contact-us/
Disallow: /it/products/contact-us/
Disallow: /cs/products/contact-us/
Disallow: /hu/products/contact-us/
Disallow: /es/products/contact-us/
Disallow: /ru/products/contact-us/
Disallow: /pl/products/contact-us/
Disallow: /pt/products/contact-us/
Disallow: /ko/products/contact-us/
Disallow: /vi/products/contact-us/
Disallow: /th/products/contact-us/
#
# Disallow all forums other than English
Disallow: /ja/forum/
Disallow: /fr/forum/
Disallow: /zh/forum/
Disallow: /de/forum/
Disallow: /it/forum/
Disallow: /cs/forum/
Disallow: /hu/forum/
Disallow: /es/forum/
Disallow: /ru/forum/
Disallow: /pl/forum/
Disallow: /pt/forum/
Disallow: /ko/forum/
Disallow: /vi/forum/
Disallow: /th/forum/
#
# Disallow unnecessary video gallery pages
# Need to add individual lines for each language not using wildcards
Disallow: /*/video-gallery/flaw-detection/
Disallow: /*/video-gallery/thickness-gaging/
Disallow: /*/video-gallery/corrosion-testing/
Disallow: /*/video-gallery/remote-visual-inspection/
Disallow: /*/video-gallery/high-speed-video/
Disallow: /*/video-gallery/industrial-scanner/
Disallow: /*/video-gallery/microscope-solutions/
Disallow: /*/video-gallery/optical-metrology/
Disallow: /*/video-gallery/integrated-inspection-systems/
Disallow: /*/video-gallery/xrf-xrd-analyzers/
User-agent: XoviBot
Disallow: /
#
User-agent: Baiduspider
Disallow: /
User-agent: 008
Disallow: /
User-agent: 200PleaseBot
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: CRAZYWEBCRAWLER
Disallow: /
User-agent: diffbot
Disallow: /
User-agent: DomainAppender
Disallow: /
User-agent: DomainSigmaCrawler
Disallow: /
User-agent: Ezooms Robot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: Google-HTTP-Java-Client
Disallow: /
User-agent: Grapeshot
Disallow: /
User-agent: Heritrix
Disallow: /
User-agent: heritrix/2.0.2 +http://www.adsafemedia.com
Disallow: /
User-agent: HTTrack 3
Disallow: /
User-agent: James BOT
Disallow: /
User-agent: Java
Disallow: /
User-agent: linkdexbot
Disallow: /
User-agent: LivelapBot/0.2 (http://site.livelap.com/crawler)
Disallow: /
User-agent: LS Session
Disallow: /
User-agent: ltx71
Disallow: /
User-agent: meanpathbot
Disallow: /
User-agent: MetaURI API/2.0 +metauri.com
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
Disallow: /
User-agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
Disallow: /
User-agent: Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: NerdyBot
Disallow: /
User-agent: netEstate NE Crawler
Disallow: /
User-agent: NetShelter ContentScan
Disallow: /
User-agent: Niki-Bot
Disallow: /
User-agent: PaperLiBot
Disallow: /
User-agent: Perl LWP
Disallow: /
User-agent: Photon
Disallow: /
User-agent: postano
Disallow: /
User-agent: proximic
Disallow: /
User-agent: Recorded Future
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SmeshBot
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Turnitin Robot
Disallow: /
User-agent: uMBot-LN
Disallow: /
User-agent: UptimeRobot/2.0
Disallow: /
User-agent: WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
Disallow: /
User-agent: WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
Disallow: /
User-agent: WiseGuys Robot
Disallow: /
User-agent: voltron
Disallow: /
User-agent: woobot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: Yahoo Pipes 1.0
Disallow: /
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Allow: /wp-content/uploads/
Disallow: /trackback
Disallow: /?ref=
Disallow: /feed/
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Disallow: /?s=
Disallow: /search
User-agent: OutfoxBot/0.5
User-agent: complex_network_group
User-agent: imparser09.yandex.ru
User-agent: spider14.yandex.ru
User-agent: yandex.ru
User-agent: yandex
User-agent: Alexibot
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Aqua_Products
User-agent: BackDoorBot
User-agent: BackDoorBot/1.0
User-agent: BPImageWalker/2.0
User-agent: Black.Hole
User-agent: BlackWidow
User-agent: BlowFish
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: Bot mailto:craftbot@yahoo.com
User-agent: BotALot
User-agent: BotRightHere
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: Copernic
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: DISCo
User-agent: DISCo Pump 3.0
User-agent: DISCo Pump 3.2
User-agent: DISCoFinder
User-agent: DittoSpyder
User-agent: Download Demon
User-agent: Download Demon/3.2.0.8
User-agent: Download Demon/3.5.0.11
User-agent: EirGrabber
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Express WebPictures
User-agent: Express WebPictures (www.express-soft.com)
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: Flaming AttackBot
User-agent: FlashGet
User-agent: FlashGet WebWasher 3.2
User-agent: Foobot
User-agent: FrontPage
User-agent: FrontPage [NC,OR]
User-agent: Gaisbot
User-agent: GetRight
User-agent: GetRight/2.11
User-agent: GetRight/3.1
User-agent: GetRight/3.2
User-agent: GetRight/3.3
User-agent: GetRight/3.3.3
User-agent: GetRight/3.3.4
User-agent: GetRight/4.0.0
User-agent: GetRight/4.1.0
User-agent: GetRight/4.1.1
User-agent: GetRight/4.1.2
User-agent: GetRight/4.2
User-agent: GetRight/4.2b (Portuguxeas)
User-agent: GetRight/4.2c
User-agent: GetRight/4.3
User-agent: GetRight/4.5
User-agent: GetRight/4.5a
User-agent: GetRight/4.5b
User-agent: GetRight/4.5b1
User-agent: GetRight/4.5b2
User-agent: GetRight/4.5b3
User-agent: GetRight/4.5b6
User-agent: GetRight/4.5b7
User-agent: GetRight/4.5c
User-agent: GetRight/4.5d
User-agent: GetRight/4.5e
User-agent: GetRight/5.0beta1
User-agent: GetRight/5.0beta2
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go!Zilla (www.gozilla.com)
User-agent: Go!Zilla 3.3 (www.gozilla.com)
User-agent: Go!Zilla 3.5 (www.gozilla.com)
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: HTTrack 3.0x
User-agent: HTTrack [NC,OR]
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Image Stripper
User-agent: Image Sucker
User-agent: Indy Library
User-agent: Indy Library [NC,OR]
User-agent: InfoNaviRobot
User-agent: InterGET
User-agent: Internet Ninja
User-agent: Internet Ninja 4.0
User-agent: Internet Ninja 5.0
User-agent: Internet Ninja 6.0
User-agent: Iron33/1.0.2
User-agent: JOC Web Spider
User-agent: JennyBot
User-agent: JetCar
User-agent: Kenjin Spider
User-agent: Kenjin.Spider
User-agent: Keyword Density/0.9
User-agent: Keyword.Density
User-agent: LNSpiderguy
User-agent: LeechFTP
User-agent: LexiBot
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan/8.1a.Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: Mass Downloader
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: Mister PiX
User-agent: Mister PiX version.dll
User-agent: Mister Pix II 2.01
User-agent: Mister Pix II 2.02a
User-agent: Mister.PiX
User-agent: NICErsPRO
User-agent: NPBot
User-agent: NPbot
User-agent: Navroad
User-agent: NearSite
User-agent: Net Vampire
User-agent: Net Vampire/3.0
User-agent: NetAnts
User-agent: NetAnts/1.10
User-agent: NetAnts/1.23
User-agent: NetAnts/1.24
User-agent: NetAnts/1.25
User-agent: NetMechanic
User-agent: NetSpider
User-agent: NetZIP
User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Explorer/1.2
User-agent: Offline Explorer/1.4
User-agent: Offline Explorer/1.6
User-agent: Offline Explorer/1.7
User-agent: Offline Explorer/1.9
User-agent: Offline Explorer/2.0
User-agent: Offline Explorer/2.1
User-agent: Offline Explorer/2.3
User-agent: Offline Explorer/2.4
User-agent: Offline Explorer/2.5
User-agent: Offline Navigator
User-agent: Offline.Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gatherer
User-agent: Oracle Ultra Search
User-agent: PageGrabber
User-agent: Papa Foto
User-agent: PerMan
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: QueryN.Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: ReGet
User-agent: RealDownload
User-agent: RealDownload/4.0.0.40
User-agent: RealDownload/4.0.0.41
User-agent: RealDownload/4.0.0.42
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2.0
User-agent: WebCopier
User-agent: WebCopier v.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac.*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com/webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Website Quester - www.asona.org
User-agent: Website Quester - www.esalesbiz.com/extra/
User-agent: Website eXtractor
User-agent: Website eXtractor (http:/www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: Wget
User-agent: Wget/1.10.2
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1+cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmos
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb/clsHTTP
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: tAkeOut
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
user-agent: AhrefsBot
disallow: /
User-agent: Yandex
Disallow: /index
User-agent: *
Disallow: /BingSiteAuth.xml
Disallow: /cgi-bin/
Disallow: /ekonomirubriker_wp.html
User-agent: 007ac9 Crawler
disallow: /
User-agent: 1 2 3 Submit PRO
disallow: /
User-agent: 200PleaseBot
disallow: /
User-agent: 2ADAMbot
disallow: /
User-agent: 2ADAMbot/1.0
disallow: /
User-agent: 360Spider
disallow: /
user-agent: Abonti
disallow: /
user-agent: Abonti/0.92
disallow: /
user-agent: abot v1.0
disallow: /
user-agent: aboutthedomain
disallow: /
user-agent: Add Catalog
disallow: /
user-agent: Add Catalog/2.1
disallow: /
user-agent: AdvBot
disallow: /
user-agent: AdvBot/2.0
disallow: /
user-agent: AhrefsBot
disallow: /
user-agent: Ahrefs-Bot
disallow: /
user-agent: AhrefsBot/1.0
disallow: /
user-agent: Ahrefs-Bot/1.0
disallow: /
user-agent: Ahrefs-Bot/2.0
disallow: /
user-agent: Ahrefs-Bot/3.0
disallow: /
user-agent: Ahrefs-Bot/4.0
disallow: /
user-agent: Ahrefs-Bot/5.0
disallow: /
user-agent: aiHitBot
disallow: /
user-agent: aiHitBot/2.9
disallow: /
user-agent: Anonymous/0.0
disallow: /
user-agent: Arachnida
disallow: /
user-agent: Associative Spider
disallow: /
User-agent: Baiduspider
disallow: /
User-agent: Baidu Spider
disallow: /
User-agent: Battleztar Bazinga
disallow: /
User-agent: Battleztar Bazinga/0.01
disallow: /
User-agent: BDFetch
disallow: /
User-agent: betaBot
disallow: /
User-agent: bieshu
disallow: /
User-agent: Bigli SEO
disallow: /
User-agent: Blackboard Safeassign
disallow: /
User-agent: Blazer 1.0
disallow: /
User-agent: BLEXBot
disallow: /
User-agent: BLEXBot/1.0
disallow: /
User-agent: BLP_bbot
disallow: /
User-agent: BLP_bbot/0.1
disallow: /
User-agent: BOIA-Accessibility-Agent/PR 1.0
disallow: /
User-agent: BOT for JCE
disallow: /
User-agent: BOT/0.1 (BOT for JCE)
disallow: /
User-agent: BPImageWalker
disallow: /
User-agent: BPImageWalker/2.0
disallow: /
User-agent: BUbiNG
disallow: /
User-agent: BuiBui-Bot
disallow: /
User-agent: BuiBui-Bot/1.0
disallow: /
User-agent: ca-crawler
disallow: /
User-agent: ca-crawler/1.0
disallow: /
User-agent: CakePHP
disallow: /
User-agent: Calypso v/0.01
disallow: /
User-agent: Calypso
disallow: /
User-agent: CB/Nutch-1.7
disallow: /
User-agent: CCBot
disallow: /
User-agent: CCBot/2.0
disallow: /
User-agent: Checkbot
disallow: /
User-agent: checkgzipcompression.com
disallow: /
User-agent: chushou
disallow: /
User-agent: CloudServerMarketSpider
disallow: /
User-agent: CloudServerMarketSpider/1.0
disallow: /
User-agent: CMS Crawler
disallow: /
User-agent: CMS Crawler: http://www.cmscrawler.com
disallow: /
User-agent: coccoc
disallow: /
User-agent: CoinCornerBot
disallow: /
User-agent: CoinCornerBot/1.1
disallow: /
User-agent: Copyscape
disallow: /
User-agent: crawler4j
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.0
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.1
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.7
disallow: /
User-agent: CrazyWebCrawler
disallow: /
User-agent: CrazyWebCrawler-Spider
disallow: /
User-agent: Crowsnest
disallow: /
User-agent: Crowsnest/0.5
disallow: /
User-agent: Curious George - www.analyticsseo.com/crawler
disallow: /
User-agent: Curious George
disallow: /
User-agent: cuwhois
disallow: /
User-agent: cuwhois/1.0
disallow: /
User-agent: dahoms
disallow: /
User-agent: datagnionbot
disallow: /
User-agent: DeuSu/5.0.2
disallow: /
User-agent: Digincore
disallow: /
User-agent: Digincore bot
disallow: /
User-agent: Dispatch/0.11.0
disallow: /
User-agent: Domain Re-Animator Bot
disallow: /
User-agent: DomainAppender /1.0
disallow: /
User-agent: DomainAppender
disallow: /
User-agent: DomainSigmaCrawler
disallow: /
User-agent: DomainSigmaCrawler/0.1
disallow: /
User-agent: Domnutch
disallow: /
User-agent: Domnutch-Bot
disallow: /
User-agent: Domnutch-Bot/Nutch
disallow: /
User-agent: Domnutch-Bot/Nutch-1.0
disallow: /
User-agent: dotbot
disallow: /
User-agent: ECCP/1.2.1
disallow: /
User-agent: eCommerceBot
disallow: /
User-agent: enlle punto com/Nutch-1.9
disallow: /
User-agent: EPiServer Link Checker
disallow: /
User-agent: EuripBot
disallow: /
User-agent: EuripBot/2.0
disallow: /
User-agent: evc/2.0
disallow: /
User-agent: evc-batch
disallow: /
User-agent: evc-batch/2.0
disallow: /
User-agent: Express WebPictures
disallow: /
User-agent: Faraday v0.8.8
disallow: /
User-agent: Faraday
disallow: /
User-agent: Findxbot
disallow: /
User-agent: Findxbot/1.0
disallow: /
User-agent: Flamingo_SearchEngine
disallow: /
User-agent: Flipboard Robot
disallow: /
User-agent: GetProxi.es-bot
disallow: /
User-agent: GetProxi.es-bot/1.1
disallow: /
User-agent: GigablastOpenSource
disallow: /
User-agent: GigablastOpenSource/1.0
disallow: /
User-agent: Girafabot
disallow: /
User-agent: Gluten Free Crawler
disallow: /
User-agent: Gluten Free Crawler/1.0
disallow: /
User-agent: GriffinBot
disallow: /
User-agent: GrifinBot/0.01
disallow: /
User-agent: GWPImages
disallow: /
User-agent: GWPImages/1.0
disallow: /
User-agent: Haiula
disallow: /
User-agent: Haiula/1.4
disallow: /
User-agent: HaosouSpider
disallow: /
User-agent: Hivemind
disallow: /
User-agent: HostHarvest
disallow: /
User-agent: HostHarvest/0.4.28
disallow: /
User-agent: HRCrawler
disallow: /
User-agent: HRCrawler/2.0
disallow: /
User-agent: http://git.io/tl_S2w
disallow: /
User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
disallow: /
User-agent: HubSpot Links Crawler 1.0
disallow: /
User-agent: HubSpot Webcrawler
disallow: /
User-agent: HubSpot
disallow: /
User-agent: hunchan
disallow: /
User-agent: HyperCrawl
disallow: /
User-agent: HyperCrawl/0.2
disallow: /
User-agent: ICAP-IOD
disallow: /
User-agent: ICC-Crawler
disallow: /
User-agent: ICC-Crawler/2.0
disallow: /
User-agent: Ichiro Robot
disallow: /
User-agent: image.coccoc/1.0
disallow: /
User-agent: Image2play
disallow: /
User-agent: Image2play/0.1
disallow: /
User-agent: Indy Library
disallow: /
User-agent: InsightsCollector
disallow: /
User-agent: InsightsCollector/0.1
disallow: /
User-agent: InsightsCollector/0.1beta
disallow: /
User-agent: integrity/5
disallow: /
User-agent: InterNaetBoten
disallow: /
User-agent: InterNaetBoten/0.99
disallow: /
User-agent: IRL Crawler
disallow: /
User-agent: James BOT - WebCrawler
disallow: /
User-agent: James BOT
disallow: /
User-agent: JamesBOT
disallow: /
User-agent: JetBrains 5.0
disallow: /
User-agent: JetBrains
disallow: /
User-agent: Kraken
disallow: /
User-agent: Kraken/0.1
disallow: /
User-agent: Kyoto-Tohoku-Crawler/v1
disallow: /
User-agent: larbin
disallow: /
User-agent: lechenie
disallow: /
User-agent: libwww-perl
disallow: /
User-agent: link checker
disallow: /
User-agent: Link/1.0
disallow: /
User-agent: linkCheck
disallow: /
User-agent: linkCheckV3.0
disallow: /
User-agent: Linkdex
disallow: /
User-agent: linkdex.com/v2.0
disallow: /
User-agent: linkdex.com/v2.1
disallow: /
User-agent: LinkdexBot
disallow: /
User-agent: linkdexbot/2.0
disallow: /
User-agent: linkdexbot/2.1
disallow: /
User-agent: linkdexbot-mobile/2.1
disallow: /
User-agent: LinkpadBot
disallow: /
User-agent: LinkpadBot/1.06
disallow: /
User-agent: LinqiaScrapeBot
disallow: /
User-agent: LinqiaScrapeBot/1.0
disallow: /
User-agent: Lipperhey SEO Service
disallow: /
User-agent: Lipperhey
disallow: /
User-agent: Lipperhey-Kaus-Australis
disallow: /
User-agent: Lipperhey-Kaus-Australis/5.0
disallow: /
User-agent: listicka
disallow: /
User-agent: LSSRocketCrawler
disallow: /
User-agent: LSSRocketCrawler/1.0 LightspeedSystems
disallow: /
User-agent: LSSRocketCrawler/1.0
disallow: /
User-agent: ltx71
disallow: /
User-agent: LWNutch/Nutch-1.4
disallow: /
User-agent: Mail.RU
disallow: /
User-agent: Mail.RU_Bot
disallow: /
User-agent: Mail.RU_Bot/2.0
disallow: /
User-agent: Mail.RU_Bot/Fast/2.0
disallow: /
User-agent: md5sum
disallow: /
User-agent: md5sum\x22
disallow: /
User-agent: meanpathbot
disallow: /
User-agent: MegaIndex.ru
disallow: /
User-agent: MegaIndex.ru/2.0
disallow: /
User-agent: mezhpozvonochnoi
disallow: /
User-agent: Mike-Crawler
disallow: /
User-agent: MixBot
disallow: /
User-agent: MixrankBot
disallow: /
User-agent: MJ12bot
disallow: /
User-agent: Monkeybot/0.1
disallow: /
User-agent: my crawler
disallow: /
User-agent: My Nutch Spider/Nutch-1.9
disallow: /
User-agent: mycrowl/Nutch-1.9
disallow: /
User-agent: MyGreatUA/2.0
disallow: /
User-agent: MyIPTest
disallow: /
User-agent: NameProtect Robot
disallow: /
User-agent: NerdyBot
disallow: /
User-agent: Netcraft Spider
disallow: /
User-agent: netEstate NE Crawler
disallow: /
User-agent: NetLyzer FastProbe
disallow: /
User-agent: NetResearchServer
disallow: /
User-agent: NetResearchServer/4.0
disallow: /
User-agent: Nmap Scripting Engine
disallow: /
User-agent: node.io
disallow: /
User-agent: node.js
disallow: /
User-agent: Node/simplecrawler 0.5.2
disallow: /
User-agent: Node/simplecrawler
disallow: /
User-agent: oBot/2.3.1
disallow: /
User-agent: omgilibot
disallow: /
User-agent: omgilibot/0.4
disallow: /
User-agent: Online Domain Tools - Online Website Link Checker
disallow: /
User-agent: Online Domain Tools - Online Website Link Checker/1.2
disallow: /
User-agent: Openfind Robot
disallow: /
User-agent: OpenHoseBot
disallow: /
User-agent: OpenHoseBot/2.1
disallow: /
User-agent: Openstat
disallow: /
User-agent: Openstat/0.1
disallow: /
User-agent: OptimizationCrawler
disallow: /
User-agent: OptimizationCrawler/0.2
disallow: /
User-agent: Page Analyzer v4.0
disallow: /
User-agent: Page Analyzer
disallow: /
User-agent: PageAnalyzer
disallow: /
User-agent: PageAnalyzer/1.1
disallow: /
User-agent: PageAnalyzer/1.5
disallow: /
User-agent: PagesInventory
disallow: /
User-agent: Pagespeed/1.1 Fetcher
disallow: /
User-agent: Pagespeed/1.1
disallow: /
User-agent: Pagespeedbot
disallow: /
User-agent: Perl LWP
disallow: /
User-agent: PHPCrawl
disallow: /
User-agent: phpSiteCheck 1.0
disallow: /
User-agent: phpSiteCheck
disallow: /
User-agent: Plukkie
disallow: /
User-agent: POGS/2.0
disallow: /
User-agent: Powermarks
disallow: /
User-agent: PowerPivot
disallow: /
User-agent: PRIVACY_ENFAQ.jsp
disallow: /
User-agent: Prlog
disallow: /
User-agent: Prlog/1.0
disallow: /
User-agent: publiclibraryarchive.org
disallow: /
User-agent: publiclibraryarchive.org/1.0
disallow: /
User-agent: Putin
disallow: /
User-agent: Putin spider
disallow: /
User-agent: qingdao
disallow: /
User-agent: QlikView
disallow: /
User-agent: quipu
disallow: /
User-agent: quipu/1.0
disallow: /
User-agent: quipu/2.0
disallow: /
User-agent: R6_CommentReader
disallow: /
User-agent: R6_FeedFetcher
disallow: /
User-agent: Riddler
disallow: /
User-agent: RivalSeek.com-Bot
disallow: /
User-agent: rogerbot
disallow: /
User-agent: rogerbot/1.0
disallow: /
User-agent: rootlink
disallow: /
User-agent: RU_Bot/2.0
disallow: /
User-agent: Scopia
disallow: /
User-agent: Scopia crawler
disallow: /
User-agent: Scopia crawler 1.0
disallow: /
User-agent: Scopia crawler 1.1
disallow: /
User-agent: Scopia crawler 1.2
disallow: /
User-agent: Scrapy
disallow: /
User-agent: Scrapy/0.16.5
disallow: /
User-agent: Scrapy/0.24.4
disallow: /
User-agent: Scrapy/0.24.5
disallow: /
User-agent: Scrapy/0.24.6
disallow: /
User-agent: Scrapy/1.0.1
disallow: /
User-agent: Screaming Frog SEO Spider
disallow: /
User-agent: Screaming Frog SEO Spider/2,55
disallow: /
User-agent: Screaming Frog SEO Spider/2.55
disallow: /
User-agent: Screaming Frog SEO Spider/3.1
disallow: /
User-agent: Screaming Frog SEO Spider/3.3
disallow: /
User-agent: Screaming Frog SEO Spider/4.1
disallow: /
User-agent: Screaming Frog SEO Spider/5.0
disallow: /
User-agent: Screaming Frog SEO Spider/5.1
disallow: /
User-agent: Screaming Frog SEO Spider/5.1 Beta 2
disallow: /
User-agent: scrutiny/4
disallow: /
User-agent: SemrushBot
disallow: /
User-agent: SemrushBot-SA
disallow: /
User-agent: SEOkicks
disallow: /
User-agent: SEOkicks-Robot
disallow: /
User-agent: SEOlyticsCrawler
disallow: /
User-agent: SEOlyticsCrawler/3.0
disallow: /
User-agent: SEOstats 2.1.0
disallow: /
User-agent: Seosys/Nutch-2.3
disallow: /
User-agent: SetCronJob/1.0
disallow: /
User-agent: SeznamBot
disallow: /
User-agent: SheerBoredom.Experimental.Robot
disallow: /
User-agent: SheerBoredom.Experimental.Robot/0.2
disallow: /
User-agent: ShowyouBot
disallow: /
User-agent: Simplecrawler
disallow: /
User-agent: SISTRIX Crawler
disallow: /
User-agent: sistrix
disallow: /
User-agent: SiteBot
disallow: /
User-agent: SiteBot/0.1
disallow: /
User-agent: SiteExplorer
disallow: /
User-agent: SiteExplorer/1.0
disallow: /
User-agent: SiteExplorer/1.0b
disallow: /
User-agent: Siteluxbot
disallow: /
User-agent: Siteluxbot/1.0
disallow: /
User-agent: SkimBot
disallow: /
User-agent: SkimBot/1.0
disallow: /
User-agent: sky nutch crawler/Nutch-1.9
disallow: /
User-agent: SMTBot
disallow: /
User-agent: SMTBot/1.0
disallow: /
User-agent: SNK Screenshot Bot
disallow: /
User-agent: SNK Screenshot Bot/0.20
disallow: /
User-agent: Sogou Spider
disallow: /
User-agent: Sogou web spider
disallow: /
User-agent: SpamBayes
disallow: /
User-agent: SpamBayes/1.1a3+
disallow: /
User-agent: spbot
disallow: /
User-agent: spbot/4.4.2
disallow: /
User-agent: spiderbot
disallow: /
User-agent: SpiderLing
disallow: /
User-agent: Spiderbot/Nutch-1.7
disallow: /
User-agent: spray-can
disallow: /
User-agent: spray-can/1.2.1
disallow: /
User-agent: SSG/3.0
disallow: /
User-agent: Statastico
disallow: /
User-agent: Statastico/4.0
disallow: /
User-agent: Steeler
disallow: /
User-agent: Steeler/3.5
disallow: /
User-agent: Stratagems Kumo
disallow: /
User-agent: Stratagems
disallow: /
User-agent: StudioFACA Search
disallow: /
User-agent: StudioFACA
disallow: /
User-agent: sukibot
disallow: /
User-agent: sukibot_heritrix
disallow: /
User-agent: sukibot_heritrix/3.1.1
disallow: /
User-agent: SurveyBot
disallow: /
User-agent: Synapse
disallow: /
User-agent: Synthesio Crawler release MonaLisa
disallow: /
User-Agent: tbot-nutch/Nutch-1.10
disallow: /
User-Agent: Traackr.com bot
disallow: /
User-Agent: trendictionbot
disallow: /
User-Agent: Trendiction-Bot
disallow: /
User-Agent: TrueBot
disallow: /
User-Agent: TrueBot/1.0
disallow: /
User-Agent: TulipChain/5.xx
disallow: /
User-Agent: TWMBot/0.1
disallow: /
User-Agent: Typhoeus
disallow: /
User-Agent: UCMore Crawler App
disallow: /
User-Agent: uMBot-LN
disallow: /
User-Agent: uMBot-LN/1.0
disallow: /
User-Agent: updown_tester
disallow: /
User-Agent: URLChecker
disallow: /
User-agent: V1.0/1.2
disallow: /
User-agent: w3af.org
disallow: /
User-agent: WASALive
disallow: /
User-agent: WASALive-Bot
disallow: /
User-agent: vBSEO
disallow: /
User-agent: WBSearchBot
disallow: /
User-agent: WBSearchBot/1.1
disallow: /
User-agent: WeAreNotEvil
disallow: /
User-agent: WebAlta
disallow: /
User-agent: WebAlta Crawler
disallow: /
User-agent: Web corpus crawler
disallow: /
User-agent: WebCookies
disallow: /
User-agent: WebCookies/1.0
disallow: /
User-agent: WebCopier vx.xa
disallow: /
User-agent: WebQL
disallow: /
User-agent: Webscout
disallow: /
User-agent: Webscout/1.0
disallow: /
User-agent: Web-sniffer
disallow: /
User-agent: Web-sniffer/1.1.0
disallow: /
User-agent: Webster Pro V3.4
disallow: /
User-agent: WebTarantula.com Crawler
disallow: /
User-agent: WeCrawlForThePeace
disallow: /
User-agent: VegeBot
disallow: /
User-agent: WeLikeLinks
disallow: /
User-agent: VeriCiteCrawler
disallow: /
User-agent: VeriCiteCrawler/Nutch-1.9
disallow: /
User-agent: WhatWeb
disallow: /
User-agent: WhatWeb/0.4.8-dev
disallow: /
User-agent: Visited by http://tools.geek-tools.org
disallow: /
User-agent: Voila Robot
disallow: /
User-agent: voltron
disallow: /
User-agent: woobot
disallow: /
User-agent: woobot/1.1
disallow: /
User-agent: woobot/2.0
disallow: /
User-agent: Vorboss Web Crawler
disallow: /
User-agent: Vorboss Web Crawler/Nutch-2.3
disallow: /
User-agent: WorldBrewBot
disallow: /
User-agent: WorldBrewBot/2.1
disallow: /
User-agent: worldwebheritage.org
disallow: /
User-agent: worldwebheritage.org/1.0
disallow: /
User-agent: wscheck.com
disallow: /
User-agent: wscheck.com/1.0.0
disallow: /
User-agent: www.deadlinkchecker.com
disallow: /
User-agent: www.petitsage.fr site detector 0.4
disallow: /
User-agent: WWW-Mechanize
disallow: /
User-agent: WWW-Mechanize/1.74
disallow: /
User-agent: Xenu Link Sleuth
disallow: /
User-agent: Xenu's Link Sleuth
disallow: /
User-agent: XoviBot
disallow: /
User-agent: XoviBot/2.0
disallow: /
User-agent: XSpider
disallow: /
User-agent: Yandex Robot
disallow: /
User-agent: Yandex
disallow: /
User-agent: Yetibot
disallow: /
User-agent: YisouSpider
disallow: /
User-agent: yoozBot
disallow: /
User-agent: yoozBot-2.2
disallow: /
User-agent: zgrab/0.x
disallow: /
User-agent: zzabmbot
disallow: /
User-agent: zzabmbot/1.0
disallow: /
#block 80legs / crawler4j spiders
User-agent: 008
User-agent: crawler4j
User-agent: crawler4j (http://code.google.com/p/crawler4j/)
User-agent: Curious George - www.analyticsseo.com/crawler
User-agent: Curious George - www.analyticsseo.com
User-agent: MegaIndex.ru/2.0
User-agent: MegaIndex.ru/
User-agent: +https://www.megaindex.ru/
User-agent: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
User-agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
User-agent: MJ12
User-agent: Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
User-agent: Steeler
User-agent: Microsoft-WebDAV-MiniRedir/6.1.7601
User-agent: Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
User-agent: Findxbot
User-agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
User-agent: SeznamBot
User-agent: Seznam
User-agent: Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
User-agent: Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
User-agent: Vagabondo
User-agent: Jakarta Commons-HttpClient/3.0.1
User-agent: LWP::Simple/5.822
User-agent: Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
User-agent: SMTBot
User-agent: Mozilla/5.0 (compatible; memoryBot/1.22.56 +http://internetmemory.org/en/)
User-agent: memoryBot
User-agent: Mozilla/5.0 (compatible; smrjbot/0.0.20)
User-agent: smrjbot
User-agent: Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
User-agent: spbot
Disallow:/
ФРГ - Кёльн - 46.163.85.226
Hosteurope GmbH
Host Europe GmbH
HTTP/1.1 200 OK
Date: Fri, 28 Sep 2018 11:15:17 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: Apache
Set-Cookie: a34cb80b9394825b1606ca77ca63670d=66327c6c44331cb79a9393590637e674; path=/; HttpOnly
P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Expires: Wed, 17 Aug 2005 00:00:00 GMT
Last-Modified: Fri, 28 Sep 2018 11:15:17 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Кнопка для анализа сайта в один клик, для установки перетащите ссылку на "Панель закладок"