Bot User Agents

> tail -n 1000000 nginx-access.log | egrep -o 'compatible; .*;' | cut -d \; -f 2-2 | sort | uniq -c | sort -h
   1  discobot/1.0
   1  Feedspotbot/1.0
   1  Konqueror/3.0-rc2
   1  Konqueror/3.1-rc2
   1  Konqueror/3.1-rc4
   1  Konqueror/3.1-rc5
   1  Konqueror/3.1-rc6
   1  Konqueror/3.5
   1  MSIE 5.0
   1  MSIE 5.5
   1  MSIE 8.0, windows NT 5.1
   1  OmniWeb/4.1-v422
   1  OptimizationCrawler/0.2
   1  Uptimebot/0.2.41
   1  WebCapture 1.0
   1  WebCapture 2.0
   1  Yahoo! Slurp China
   1  YodaoBot/1.0
   2  DomainSigmaCrawler/0.1
   2  MegaIndex.ru/2.0
   2  MSIE 10.6
   2  Qwantify/2.2w
   2  Uptimebot/0.2.43
   4  Let's Encrypt validation server
   4  MixrankBot
   4  Twingly Recon
   5  MSIE8.0
   5  Qwantify/2.3w
   8  FlipboardRSS/1.2
  11  archive.org_bot
  12  DuckDuckGo-Favicons-Bot/1.0
  13  Gluten Free Crawler/1.0
  13  MSIE 5.01
  14  YandexImages/3.0
  15  spbot/5.0.3
  18  MSIE 9.11
  18  spbot/5.0.2
  21  Uptimebot/1.0
  22  NetcraftSurveyAgent/1.0
  31  WBSearchBot/1.1
  36  SeznamBot/3.2
  38  Nmap Scripting Engine
  48  DotBot/1.1
  50  Dataprovider/6.101
  77  MSIE or Firefox mutant
  79  MojeekBot/0.6
  85  theoldreader.com
 120  SMTBot/1.0
 154  FlipboardRSS/1.1
 173  Exabot/3.0
 196  Googlebot-Mobile/2.1
 216  Linux x86_64
 311  linkdexbot/2.0
 361  MSIE 10.0
 447    MSIE   7.0
 481  aiHitBot/2.9
 502  SiteExplorer/1.1b
 750  MSIE 7.0
 767  MJ12bot/v1.4.7
 787  MJ12bot/v1.4.6
 835  Windows
 861  MSIE 8.0
 981  MSIE 6.0
1215  YandexMobileBot/3.0
1953  SiteExplorer/1.0b
2518  HaosouSpider
2579  Yahoo! Slurp
3017  YandexBot/3.0
6761  MSIE 9.0
8005  BLEXBot/1.0
9124  Baiduspider/2.0
12091  bingbot/2.0
23582  Googlebot/2.1
35585  MJ12bot/v1.4.5
88015  AhrefsBot/5.1
89224  XoviBot/2.0
255831  SemrushBot/1.1~bl

So, SemrushBot, XoviBot and AhrefsBot are banned.