Want only Google to crawl your site? This configuration blocks all bots by default and explicitly allows only Googlebot. Useful for new sites, staging environments, or sites that only care about Google traffic.
Some sites get 90%+ traffic from Google and want to reduce server load from other crawlers. Or you may want to prevent Bing, Yandex, and AI bots from accessing your content entirely.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to Robots.txt — Disallow All Bots Except Googlebot
- Add User-agent: * with Disallow: / (blocks everything by default)
- Add User-agent: Googlebot with Allow: / (overrides the block for Google)
- Add your sitemap URL
- Generate and verify the output
- Upload to your site root
Pro Tips
- Order matters: specific user-agent rules override the wildcard (*) rule
- Consider also allowing Bingbot — Bing drives 5-10% traffic for some niches
- This blocks social media previews too (Facebook, Twitter) — add their bots if needed
- Test with Google Search Console robots.txt tester after uploading
Frequently Asked Questions
Will this hurt my SEO?
No, as long as Googlebot is explicitly allowed. Google will crawl and index your site normally. Other search engines (Bing, Yandex) will not index you.
Does blocking all bots block social media previews?
Yes. If you share links on Facebook/Twitter, their bots need access to generate previews. Add Facebookbot and Twitterbot to the allow list if needed.
How do I test if my robots.txt works?
Use Google Search Console > Settings > robots.txt report. Or visit yourdomain.com/robots.txt directly to verify the file is accessible.
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator