Recommended Crawl Rate for Bots

You can set your desired bot crawl delay in your robots.txt file by adding this after the user-agent field: Crawl-Delay: 10 That will cause any legitimate robot to wait 10 seconds between requests as they crawl your site for links. My recommendation, however, is not to set a crawl delay at all. You want bots … Read more