Trouble with BOTs

From The System Administrator Zone
Revision as of 22:52, 10 August 2017 by Sakshale (talk | contribs) (Google BOT)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

BOTs can cause serious system load issues.

Google BOT

Google does not follow the instructions set in the robots.txt file. If you are having system load problems related to Goggle BOT, you need to reduce the Google crawl rate to a minimum to lower the resource usage. You do that from your Google Search Console.

See: Change Googlebot crawl rate