Google Search Console Crawl Rate Settings

Your site is being crawled at a rate calculated as optimal by Google. You can change the crawl rate only by filing a special request using the form mentioned in the “learn more” documentation.

For some sites, Google will allow users to adjust the crawl rate and thus manually limit the amount of traffic Google has on your servers. This customization option is only available to some sites. I’ve been unable to determine what metric limits users to adjusting this option. Sites with high and low traffic have the option both available and not available (replaced by the above message).

Google tries to crawl as many of your site’s pages as we can without overwhelming your server’s bandwidth. If Google’s crawlers are slowing your site, you can change the crawl rate (the speed of Google’s requests). This feature is only available for sites at the root or subdomain level.

If you think Googlebot is crawling your site too quickly, and you want to slow it down but cannot (because the WMT option is disabled), you can file a request here to report Googlebot issues with crawling. You’ll need to know the following information before you submit a crawl-issue request.

  1. From what IP addresses are you seeing Googlebot activity?
  2. For which user-agent are you seeing Googlebot activity?
  3. How many times a day does Googlebot access your site?
  4. Additional details (Please also include a portion of the weblog that shows Google accesses so we can track down the problem quickly):

Report a problem with how Googlebot crawls your site.
You can report problems only for domain-level properties (for example, “www.example.com/”)

The rate at which Google crawls your page depends on many factors:

  • The URLs we already know about
  • Links from other web pages (within your site and on other sites)
  • URLs listed in your Sitemap.

For most sites, Googlebot shouldn’t access your site more than once every few seconds on average. However, due to network delays, it’s possible that the rate will appear to be slightly higher over short periods. If you are seeing a particular issue with the Googlebot, please share it in the comments!

Leave a Comment