Robots.txt Generator

What is robots.txt generator?

A robots.txt file tells crawlers which parts of your site they may request. It is a request, not a lock, so it is for crawl control and not for hiding private data.

This tool writes a correct file and, if you like, adds rules for popular AI training crawlers.

How to use the robots.txt generator

1. List the paths to block.
2. Choose whether to allow or block AI crawlers.
3. Add your sitemap and copy the file to the root of your domain.

How the robots.txt generator works

Rules are written in the standard format under User agent star. Blocked AI bots each get their own group with Disallow slash.

Worked example

Common uses of the robots.txt generator

• Launching a new site and keeping admin or cart pages out of crawl paths.
• Pointing crawlers to your XML sitemap.
• Saving crawl budget on large sites with filter pages.
• Setting a policy for AI crawlers.
• Keeping staging areas from being crawled.

Tips and common mistakes

Never block CSS or JavaScript files that pages need for rendering.

Blocking a page in robots.txt does not remove it from search. Use noindex for that.

Background: what to know before you rely on it

The robots exclusion protocol dates back to 1994. A robots.txt file is a polite request, and well behaved crawlers follow it, but it is not security. Pages that are blocked can still appear in search results if other sites link to them, only without a description. To keep a page out of search, use a noindex meta tag and let the crawler fetch it. Be careful not to block resources like CSS and JavaScript, which search engines need to render a page. A single stray slash can block an entire site, so always test the file in Google Search Console after you upload it.

Private, free and fast

This tool runs entirely in your browser. What you type or choose is not uploaded to a server and is not saved after you close the page, which keeps it fast and private. It is free to use as often as you like. See more in all seo tools, or browse the full list of tools.

Related tools

Results are for information and education. For legal, financial, medical or security decisions, check with a qualified professional. Validate markup with search engine testing tools before publishing.