Staging and development sites should NEVER be indexed by search engines. A single indexed staging page can cause duplicate content penalties on your production site. This generates a complete block for all crawlers.
Developers often forget to block staging sites. Google finds them through DNS records, linked resources, or sitemap references. One robots.txt file prevents all accidental indexing.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to Robots.txt for Staging & Development Sites
- Set User-agent: * (all bots)
- Set Disallow: / (blocks entire site)
- Do NOT add a sitemap URL
- Generate and upload to staging root
- Also add meta name robots content noindex as backup
Pro Tips
- Use BOTH robots.txt AND meta noindex for maximum protection
- Password-protect staging with HTTP Basic Auth as additional layer
- Remove the block when you go live — do not forget!
- Check Google Search Console for any accidentally indexed staging URLs
Frequently Asked Questions
Is robots.txt enough to prevent staging indexing?
Usually yes, but not guaranteed. For maximum protection, also add meta noindex tag and password protection (HTTP Basic Auth).
What if my staging site is already indexed?
Add robots.txt + meta noindex, then request removal via Google Search Console > Removals tool. It takes 1-2 days.
Should I use robots.txt or meta noindex?
Use both. robots.txt prevents crawling (saves bandwidth). Meta noindex prevents indexing even if a bot ignores robots.txt.
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator