*

*

0/500

UA for crawlers

“Crawler” is a generic term for any program (such as a robot or spider) used to automatically discover and scan the pages of a website. You can block search engine crawlers from your site by setting the UA keywords as follows.

Web Crawler Keyword Example
Googlebot Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Bingbot Bingbot Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)
Yahoo! Slurp Bot Slurp Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
DuckDuckBot duckduckbot DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
Baiduspider Baiduspider Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
YandexBot YandexBot Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Sogou Spider Sogou web spider Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Exabot Exabot Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
MJ12bot MJ12bot Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
AdsBot AdsBot AdsBot-Google (+http://www.google.com/adsbot.html)

Hits: 1124