Aggressive bots can overload your server with too many requests per second. The crawl-delay directive tells bots to wait between requests. This generator helps you set appropriate delays for different bots.
Shared hosting and small VPS servers can struggle when multiple bots crawl simultaneously. Crawl-delay of 10 seconds means a bot makes only 6 requests per minute instead of hundreds.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to Add Crawl-Delay to Robots.txt — Reduce Server Load
- Identify which bots are causing high server load (check access logs)
- Select the bot user agent (Bingbot, Yandex, etc.)
- Set Crawl-delay value (10 = 10 seconds between requests)
- Keep Googlebot WITHOUT crawl-delay (Google ignores it anyway)
- Generate and upload your robots.txt
Pro Tips
- Googlebot ignores crawl-delay — use Google Search Console crawl rate settings instead
- Bingbot and Yandex respect crawl-delay
- Crawl-delay: 10 means 1 request every 10 seconds (6/minute)
- If server is still overloaded, consider blocking non-essential bots entirely
- Check server access logs to identify the most aggressive crawlers
Frequently Asked Questions
Does Googlebot respect crawl-delay?
No. Google ignores the crawl-delay directive. To control Googlebot crawl rate, use Google Search Console > Settings > Crawl rate.
What crawl-delay value should I use?
Start with 10 (one request per 10 seconds). If server is still stressed, increase to 30. For powerful servers, 5 is fine.
Will crawl-delay hurt my SEO?
Not for Google (they ignore it). For Bing, a high crawl-delay means slower indexing of new pages. Keep it reasonable (10-30 seconds).
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator