Böse Bots und Crawler aussperren

Ihr könnt auch mithilfe der htaccess böse Bots und Crawler aussperren. Diese verbrauchen nur Bandbreite und können gefahrlos über die htaccess gesperrt werden. Hier eine Liste, welche fortlaufend aktualisiert wird:

umleitung Bild

##### Ein wenig Bandbreite und unangenehme Crawler aussperren

SetEnvIf user-agent "WebBandit" stayout=1
SetEnvIf user-agent "webbandit" stayout=1
SetEnvIf user-agent "Acunetix" stayout=1
SetEnvIf user-agent "binlar" stayout=1
SetEnvIf user-agent "BlackWidow" stayout=1
SetEnvIf user-agent "Bolt 0" stayout=1
SetEnvIf user-agent "BOT for JCE" stayout=1
SetEnvIf user-agent "casper" stayout=1
SetEnvIf user-agent "checkprivacy" stayout=1
SetEnvIf user-agent "ChinaClaw" stayout=1
SetEnvIf user-agent "clshttp" stayout=1
SetEnvIf user-agent "cmsworldmap" stayout=1
SetEnvIf user-agent "comodo" stayout=1
SetEnvIf user-agent "Custo" stayout=1
SetEnvIf user-agent "Default Browser 0" stayout=1
SetEnvIf user-agent "diavol" stayout=1
SetEnvIf user-agent "DIIbot" stayout=1
SetEnvIf user-agent "DISCo" stayout=1
SetEnvIf user-agent "dotbot" stayout=1
SetEnvIf user-agent "Download Demon" stayout=1
SetEnvIf user-agent "eCatch" stayout=1
SetEnvIf user-agent "EirGrabber" stayout=1
SetEnvIf user-agent "EmailCollector" stayout=1
SetEnvIf user-agent "EmailSiphon" stayout=1
SetEnvIf user-agent "EmailWolf" stayout=1
SetEnvIf user-agent "Express WebPictures" stayout=1
SetEnvIf user-agent "extract" stayout=1
SetEnvIf user-agent "ExtractorPro" stayout=1
SetEnvIf user-agent "EyeNetIE" stayout=1
SetEnvIf user-agent "feedfinder" stayout=1
SetEnvIf user-agent "FHscan" stayout=1
SetEnvIf user-agent "FlashGet" stayout=1
SetEnvIf user-agent "flicky" stayout=1
SetEnvIf user-agent "GetRight" stayout=1
SetEnvIf user-agent "GetWeb!" stayout=1
SetEnvIf user-agent "Go-Ahead-Got-It" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "grab" stayout=1
SetEnvIf user-agent "GrabNet" stayout=1
SetEnvIf user-agent "Grafula" stayout=1
SetEnvIf user-agent "harvest" stayout=1
SetEnvIf user-agent "HMView" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "Image Stripper" stayout=1
SetEnvIf user-agent "Image Sucker" stayout=1
SetEnvIf user-agent "InterGET" stayout=1
SetEnvIf user-agent "Internet Ninja" stayout=1
SetEnvIf user-agent "InternetSeer.com" stayout=1
SetEnvIf user-agent "jakarta" stayout=1
SetEnvIf user-agent "Java" stayout=1
SetEnvIf user-agent "JetCar" stayout=1
SetEnvIf user-agent "JOC Web Spider" stayout=1
SetEnvIf user-agent "kmccrew" stayout=1
SetEnvIf user-agent "larbin" stayout=1
SetEnvIf user-agent "LeechFTP" stayout=1
SetEnvIf user-agent "libwww" stayout=1
SetEnvIf user-agent "Mass Downloader" stayout=1
SetEnvIf user-agent "Maxthon$" stayout=1
SetEnvIf user-agent "microsoft.url" stayout=1
SetEnvIf user-agent "MIDown tool" stayout=1
SetEnvIf user-agent "miner" stayout=1
SetEnvIf user-agent "Mister PiX" stayout=1
SetEnvIf user-agent "NEWT" stayout=1
SetEnvIf user-agent "MSFrontPage" stayout=1
SetEnvIf user-agent "Navroad" stayout=1
SetEnvIf user-agent "NearSite" stayout=1
SetEnvIf user-agent "Net Vampire" stayout=1
SetEnvIf user-agent "NetAnts" stayout=1
SetEnvIf user-agent "NetSpider" stayout=1
SetEnvIf user-agent "NetZIP" stayout=1
SetEnvIf user-agent "nutch" stayout=1
SetEnvIf user-agent "Octopus" stayout=1
SetEnvIf user-agent "Offline Explorer" stayout=1
SetEnvIf user-agent "Offline Navigator" stayout=1
SetEnvIf user-agent "PageGrabber" stayout=1
SetEnvIf user-agent "Papa Foto" stayout=1
SetEnvIf user-agent "pavuk" stayout=1
SetEnvIf user-agent "pcBrowser" stayout=1
SetEnvIf user-agent "PeoplePal" stayout=1
SetEnvIf user-agent "planetwork" stayout=1
SetEnvIf user-agent "psbot" stayout=1
SetEnvIf user-agent "purebot" stayout=1
SetEnvIf user-agent "pycurl" stayout=1
SetEnvIf user-agent "RealDownload" stayout=1
SetEnvIf user-agent "ReGet" stayout=1
SetEnvIf user-agent "Rippers 0" stayout=1
SetEnvIf user-agent "SeaMonkey$" stayout=1
SetEnvIf user-agent "sitecheck.internetseer.com" stayout=1
SetEnvIf user-agent "SiteSnagger" stayout=1
SetEnvIf user-agent "skygrid" stayout=1
SetEnvIf user-agent "SmartDownload" stayout=1
SetEnvIf user-agent "sucker" stayout=1
SetEnvIf user-agent "SuperBot" stayout=1
SetEnvIf user-agent "SuperHTTP" stayout=1
SetEnvIf user-agent "Surfbot" stayout=1
SetEnvIf user-agent "tAkeOut" stayout=1
SetEnvIf user-agent "Teleport Pro" stayout=1
SetEnvIf user-agent "Toata dragostea mea pentru diavola" stayout=1
SetEnvIf user-agent "turnit" stayout=1
SetEnvIf user-agent "vikspider" stayout=1
SetEnvIf user-agent "VoidEYE" stayout=1
SetEnvIf user-agent "Web Image Collector" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "WebAuto" stayout=1
SetEnvIf user-agent "WebCopier" stayout=1
SetEnvIf user-agent "WebFetch" stayout=1
SetEnvIf user-agent "WebGo IS" stayout=1
SetEnvIf user-agent "WebLeacher" stayout=1
SetEnvIf user-agent "WebReaper" stayout=1
SetEnvIf user-agent "WebSauger" stayout=1
SetEnvIf user-agent "Website eXtractor" stayout=1
SetEnvIf user-agent "Website Quester" stayout=1
SetEnvIf user-agent "WebStripper" stayout=1
SetEnvIf user-agent "WebWhacker" stayout=1
SetEnvIf user-agent "WebZIP" stayout=1
SetEnvIf user-agent "Wget" stayout=1
SetEnvIf user-agent "Widow" stayout=1
SetEnvIf user-agent "WWW-Mechanize" stayout=1
SetEnvIf user-agent "WWWOFFLE" stayout=1
SetEnvIf user-agent "Xaldon WebSpider" stayout=1
SetEnvIf user-agent "Yandex" stayout=1
SetEnvIf user-agent "Zeus" stayout=1
SetEnvIf user-agent "zmeu" stayout=1
SetEnvIf user-agent "CazoodleBot" stayout=1
SetEnvIf user-agent "discobot" stayout=1
SetEnvIf user-agent "ecxi" stayout=1
SetEnvIf user-agent "GT::WWW" stayout=1
SetEnvIf user-agent "heritrix" stayout=1
SetEnvIf user-agent "HTTP::Lite" stayout=1
SetEnvIf user-agent "HTTrack" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "id-search" stayout=1
SetEnvIf user-agent "id-search.org" stayout=1
SetEnvIf user-agent "IDBot" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "IRLbot" stayout=1
SetEnvIf user-agent "ISC Systems iRc Search 2.1" stayout=1
SetEnvIf user-agent "LinksManager.com_bot" stayout=1
SetEnvIf user-agent "linkwalker" stayout=1
SetEnvIf user-agent "lwp-trivial" stayout=1
SetEnvIf user-agent "MFC_Tear_Sample" stayout=1
SetEnvIf user-agent "Microsoft URL Control" stayout=1
SetEnvIf user-agent "Missigua Locator" stayout=1
SetEnvIf user-agent "panscient.com" stayout=1
SetEnvIf user-agent "PECL::HTTP" stayout=1
SetEnvIf user-agent "PHPCrawl" stayout=1
SetEnvIf user-agent "PleaseCrawl" stayout=1
SetEnvIf user-agent "SBIder" stayout=1
SetEnvIf user-agent "Snoopy" stayout=1
SetEnvIf user-agent "Steeler" stayout=1
SetEnvIf user-agent "URI::Fetch" stayout=1
SetEnvIf user-agent "urllib" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "webalta" stayout=1
SetEnvIf user-agent "WebCollage" stayout=1
SetEnvIf user-agent "Wells Search II" stayout=1
SetEnvIf user-agent "WEP Search" stayout=1
SetEnvIf user-agent "zermelo" stayout=1
SetEnvIf user-agent "ZyBorg" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "libwww-perl" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "TurnitinBot" stayout=1
<IfModule !mod_authz_core.c>
deny from env=stayout
</IfModule>
<IfModule mod_authz_core.c>
<RequireAll>
Require all granted
Require not env stayout
</RequireAll>
</IfModule>
#####

Das könnt ihr einfach in eurer htaccess eintragen. Danach bekommen die Bots bzw Crawler keinen Zugriff mehr auf eure Seite und ihr spart sogar damit Bandbreite eurer Webseite.