AI companies like OpenAI, Anthropic, and Common Crawl use bots to scrape websites for training data. You can block them using robots.txt rules. This generator creates the exact rules needed to stop AI crawlers while keeping Google and Bing access.
If you do not want your content used to train AI models (ChatGPT, Claude, etc.), you need to explicitly block their crawlers. These bots respect robots.txt — blocking them is effective and immediate.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to Block AI Bots in Robots.txt — Protect Your Content
- Select GPTBot from the user agent dropdown
- Set Disallow: / (blocks entire site)
- Repeat for ClaudeBot, CCBot, Google-Extended, Bytespider
- Keep Googlebot and Bingbot allowed (for search rankings)
- Generate and copy the complete robots.txt
- Upload to your site root
Pro Tips
- GPTBot = OpenAI (ChatGPT), ClaudeBot = Anthropic (Claude)
- CCBot = Common Crawl (used by many AI companies)
- Google-Extended = Google AI training (separate from Googlebot search)
- Bytespider = ByteDance/TikTok AI crawler
- These bots DO respect robots.txt — blocking works immediately
Frequently Asked Questions
Which AI bots should I block?
The main ones: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google AI), Bytespider (ByteDance), FacebookBot (Meta AI). Block all for maximum protection.
Will blocking AI bots affect my Google ranking?
No. Googlebot (search) and Google-Extended (AI training) are separate. Blocking Google-Extended does not affect your search rankings at all.
Do AI bots actually respect robots.txt?
Yes, major AI companies (OpenAI, Anthropic, Google) have committed to respecting robots.txt. Smaller scrapers may not, but the big ones do.
Can I block AI bots but allow search engines?
Yes, that is exactly what this tool does. You set Disallow for AI bots and Allow for Googlebot/Bingbot separately.
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator