A robots.txt file tells search engines which pages to crawl and which to skip. Every website needs one. This guide explains the syntax, common rules, and best practices — plus a free generator to create yours in seconds.
Without a robots.txt file, search engines crawl everything — including admin panels, duplicate pages, and private content. A properly configured file improves crawl efficiency and prevents sensitive URLs from appearing in search results.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to How to Create a Robots.txt File — Complete Guide
- Create a plain text file named robots.txt
- Add User-agent line (which bot the rules apply to)
- Add Disallow lines (which paths to block)
- Add Allow lines (exceptions to blocks)
- Add Sitemap line (URL to your XML sitemap)
- Upload to your website root (accessible at yourdomain.com/robots.txt)
- Test with Google Search Console robots.txt tester
Pro Tips
- robots.txt MUST be at the root: yourdomain.com/robots.txt (not in a subfolder)
- Use User-agent: * to apply rules to all bots
- Disallow: / blocks the entire site; Disallow: (empty) allows everything
- Each rule goes on its own line — no commas or semicolons
- robots.txt is case-sensitive: /Admin/ and /admin/ are different
- Always end with a Sitemap: line pointing to your XML sitemap
Frequently Asked Questions
What is robots.txt?
A plain text file at your website root that tells search engine crawlers which URLs they can and cannot access. It is part of the Robots Exclusion Protocol standard.
Is robots.txt mandatory?
No, but highly recommended. Without it, bots crawl everything — wasting server resources and potentially indexing private pages.
Does robots.txt hide pages from Google?
No! It prevents crawling, not indexing. If other sites link to a blocked page, Google may still show it in results (without a snippet). Use meta noindex to prevent indexing.
Can I use wildcards in robots.txt?
Yes. Google and Bing support * (match anything) and $ (end of URL). Example: Disallow: /*.pdf$ blocks all PDF files.
How quickly does robots.txt take effect?
Google typically re-checks robots.txt every 24 hours. Changes take effect within 1-2 days for most sites.
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator