Robots txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots txt Generator

→ What is Robots.txt in SEO?

A robots.txt file informs search engine bots not to crawl specific pages or sections of a website.

The robots.txt file is part of the robots exclusion protocol (REP), a set of web standards that control how robots crawl the web, access and index content, and deliver it to users.

→ Structure of Robots.txt file

In a robots file, you're likely to encounter 5 common terms :

User agent -

The web crawler to which you are passing crawl instructions (usually a search engine bot).

Disallow -

It used to instruct a user-agent not to crawl a specific URL. For each URL, only one "Disallow:" line is allowed.

Allow (Only for Googlebot) -

The command instructs Googlebot that it may access a page or subfolder even if its parent page or subfolder is disallowed.

Crawl delay -

How much time should a crawler wait before loading and crawling page content.

Sitemap -

It is used to specify the location of any XML sitemap(s) associated with this URL.

→ How to check Robots.txt file?

Simply enter your root domain followed by /robots.txt at the end of the URL. For example, Amazon's robots file can be found at amazon.com/robots.txt.

If no .txt file is displayed, you do not presently have a live robots.txt file.

→ How to use this Robots.txt Generator tool?

The process of using this tool requires your potency in order to understand a few facts.

Step #1 - You will see a few options on the robots txt generator tool webpage; not all options are required, but you must choose carefully. The first row shows default values for all robots and whether or not to keep a crawl-delay. If you don't want to change anything, just leave them as default.

Step #2 - The second row is about sitemaps; ensure you possess one and include it in the robots.txt file.

Step #3 - Following that, you can select whether or not you want search engine bots to crawl your site.

Step #4 - The final option is disallowing, which prevents crawlers from indexing certain areas of the website.

If you want to exclude a page, write "Disallow: the link you don't want the bots to visit," and the same is true for the allowing attribute. If you believe that is all there is to the robots.txt file, you are mistaken; one incorrect line can prevent your page from being indexed. So, leave the task to the professionals and let our Robots.txt generator handle the file for you.