Robots.txt Generator

Default - All Robots are:

Crawl-Delay:

Sitemap: (leave blank if you don't have)

Search Robots:	Google
	Google Image
	Google Mobile
	MSN Search
	Yahoo
	Yahoo MM
	Yahoo Blogs
	Ask/Teoma
	GigaBlast
	DMOZ Checker
	Nutch
	Alexa/Wayback
	Baidu
	Naver
	MSN PicSearch

Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

What is Robots.txt Generator?

Enter more information about the Robots.txt Generator tool!

Robot.Txt generator

Robot.txt is a file of a set of instructions for web crawlers. The robot.txt contains instructions for web crawlers and tells them which webpage to crawl and which not to. The robot.txt file is the first file search engines look for, if not found, then there is a huge chance that search engine will not index all pages of your website.

To create robot.txt using Webtoolsz’s Robot.txt Generator tool, the user has to enter a URL of their website and enable/disable search robots, and click on ‘create robot.txt’. Within a few seconds, you will have your robot.txt ready. You can copy it or download for later use.

Keywords: web crawlers, Webtoolsz’s Robot.txt Generator, Robot.txt, how to generate robot.txt, create robot,txt, robot.txt generator, robot.txt creator.