> tail -n 1000000 nginx-access.log | egrep -o 'compatible; .*;' | cut -d \; -f 2-2 | sort | uniq -c | sort -h 1 discobot/1.0 1 Feedspotbot/1.0 1 Konqueror/3.0-rc2 1 Konqueror/3.1-rc2 1 Konqueror/3.1-rc4 1 Konqueror/3.1-rc5 1 Konqueror/3.1-rc6 1 Konqueror/3.5 1 MSIE 5.0 1 MSIE 5.5 1 MSIE 8.0, windows NT 5.1 1 OmniWeb/4.1-v422 1 OptimizationCrawler/0.2 1 Uptimebot/0.2.41 1 WebCapture 1.0 1 WebCapture 2.0 1 Yahoo! Slurp China 1 YodaoBot/1.0 2 DomainSigmaCrawler/0.1 2 MegaIndex.ru/2.0 2 MSIE 10.6 2 Qwantify/2.2w 2 Uptimebot/0.2.43 4 Let's Encrypt validation server 4 MixrankBot 4 Twingly Recon 5 MSIE8.0 5 Qwantify/2.3w 8 FlipboardRSS/1.2 11 archive.org_bot 12 DuckDuckGo-Favicons-Bot/1.0 13 Gluten Free Crawler/1.0 13 MSIE 5.01 14 YandexImages/3.0 15 spbot/5.0.3 18 MSIE 9.11 18 spbot/5.0.2 21 Uptimebot/1.0 22 NetcraftSurveyAgent/1.0 31 WBSearchBot/1.1 36 SeznamBot/3.2 38 Nmap Scripting Engine 48 DotBot/1.1 50 Dataprovider/6.101 77 MSIE or Firefox mutant 79 MojeekBot/0.6 85 theoldreader.com 120 SMTBot/1.0 154 FlipboardRSS/1.1 173 Exabot/3.0 196 Googlebot-Mobile/2.1 216 Linux x86_64 311 linkdexbot/2.0 361 MSIE 10.0 447 MSIE 7.0 481 aiHitBot/2.9 502 SiteExplorer/1.1b 750 MSIE 7.0 767 MJ12bot/v1.4.7 787 MJ12bot/v1.4.6 835 Windows 861 MSIE 8.0 981 MSIE 6.0 1215 YandexMobileBot/3.0 1953 SiteExplorer/1.0b 2518 HaosouSpider 2579 Yahoo! Slurp 3017 YandexBot/3.0 6761 MSIE 9.0 8005 BLEXBot/1.0 9124 Baiduspider/2.0 12091 bingbot/2.0 23582 Googlebot/2.1 35585 MJ12bot/v1.4.5 88015 AhrefsBot/5.1 89224 XoviBot/2.0 255831 SemrushBot/1.1~bl
So, SemrushBot, XoviBot and AhrefsBot are banned.
