Trouble with BOTs

From The System Administrator Zone


BOTs can cause serious system load issues.

Google BOT

Google does not follow the instructions set in the robots.txt file. If you are having system load problems related to Goggle BOT, you need to reduce the Google crawl rate to a minimum to lower the resource usage. You do that from your Google Search Console.

See: Change Googlebot crawl rate