WordPress sites need a properly configured robots.txt to prevent search engines from crawling admin pages, login URLs, and duplicate content. This generator creates WordPress-optimized rules you can copy-paste — no plugin required.
WordPress generates a virtual robots.txt but it lacks custom rules. Blocking /wp-admin/, /wp-includes/, and /cart/ prevents crawl budget waste and keeps private URLs out of search results.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to Generate Robots.txt for WordPress — No Plugin Needed
- Select Googlebot and Bingbot as user agents
- Add Disallow rules: /wp-admin/, /wp-includes/, /cart/, /checkout/
- Add Allow rule: /wp-admin/admin-ajax.php (needed for some themes)
- Enter your sitemap URL (usually /sitemap_index.xml or /sitemap.xml)
- Click Generate and copy the output
- Upload to your WordPress root via FTP or file manager
Pro Tips
- Never block /wp-content/uploads/ — your images need to be crawled
- If using Yoast SEO, your sitemap is at /sitemap_index.xml
- Block /feed/ only if you do not want RSS indexed
- Add crawl-delay: 10 for aggressive bots to reduce server load
Frequently Asked Questions
Where do I put robots.txt in WordPress?
Upload it to your site root (same folder as wp-config.php). Access it at yourdomain.com/robots.txt. If using cPanel, use File Manager to upload.
Does WordPress have a default robots.txt?
Yes, WordPress generates a virtual one that blocks /wp-admin/. But it is basic — a custom file gives you full control over crawl rules.
Should I block wp-admin in robots.txt?
Yes, but always allow /wp-admin/admin-ajax.php — many themes and plugins need it for frontend functionality.
Do I need a plugin for robots.txt?
No. A static robots.txt file in your root works perfectly. Plugins add unnecessary overhead for something this simple.
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator