Next.js apps expose internal routes (/_next/, /api/) that should not be indexed. A proper robots.txt prevents search engines from wasting crawl budget on framework files while keeping your actual pages fully accessible.
Next.js serves static assets from /_next/static/ and API endpoints from /api/. These should never appear in search results. A robots.txt file prevents this and improves crawl efficiency.
Try It Now — Free, No Sign-up
Open the tool and get started instantly. No sign-up, no installation needed.
Open Robots.txt Generator Now100% browser-based • No upload to server • No sign-up required
How to Generate Robots.txt for Next.js Applications
- Select All bots (*) as user agent
- Add Disallow: /_next/, /api/
- Add Disallow for any auth routes: /auth/, /login, /signup
- Allow all public pages (default)
- Add sitemap URL (usually /sitemap.xml)
- Save as public/robots.txt in your Next.js project
Pro Tips
- In Next.js, robots.txt goes in the /public folder
- Next.js 13+ App Router: you can use app/robots.ts for dynamic generation
- Never block /_next/image — your optimized images need to be crawlable
- Block /api/ routes to prevent internal endpoints from being indexed
Frequently Asked Questions
Where does robots.txt go in Next.js?
Place it in the /public folder. It will be served at yourdomain.com/robots.txt automatically. No configuration needed.
Should I block /_next/ in robots.txt?
Block /_next/data/ and /_next/static/ but NOT /_next/image — image optimization URLs should be crawlable for image search.
Can I generate robots.txt dynamically in Next.js?
Yes, in App Router (Next.js 13+) create app/robots.ts that exports a function. For Pages Router, use a custom API route or static file.
Related Tools & Guides
Ready to Use Robots.txt Generator?
Free, instant, and 100% private. No sign-up needed.
Open Robots.txt Generator