What does a robots.txt generator do?
A robots.txt file sits at the root of your domain, for example https://example.com/robots.txt. Crawlers read it before they fetch your pages. It is a set of requests, written as groups of lines: a User-agent line names the crawler, and the lines below it say which paths that crawler should skip. Writing it by hand is simple until you have several groups, and then a stray space or a missing blank line can change the meaning.
This robots.txt generator builds the file from a form. Each Bot Rules block has a User-agent dropdown, a Permission dropdown and a box for paths. You can add as many blocks as you like, enter one Sitemap URL, set an optional Crawl Delay, and press Generate Robots.txt. The output appears below, with Copy and Download buttons.
Know what it does not do. It cannot read your current robots.txt, it does not check that your paths exist, and the custom option writes Disallow lines only, with no Allow exceptions. A robots.txt file also does not protect private pages. It asks polite crawlers to stay away, and anyone can still open the URL.
How to create a robots.txt file with this tool
- In the first Bot Rules block, choose a User-agent. Pick
*to cover every crawler that does not have its own group. - Set Permission to Allow all, Disallow all or Custom paths.
- For Custom paths, type one path per line in Disallow paths, such as
/admin/or/search?. The box is ignored for the other two options. - Click + Add Another Bot Rule if a second crawler needs different treatment, for example a block for GPTBot while everyone else is allowed.
- Enter your full Sitemap URL (including https://). Add a Crawl Delay only if you really need one.
- Click Generate Robots.txt, check the output, then use Copy or Download and upload the file to the root of your domain.
What this robots.txt generator includes
Seven crawler choices
The User-agent dropdown offers * (all bots), Googlebot, Bingbot, YandexBot, GPTBot, CCBot and anthropic-ai, so search engines and AI-training crawlers can get different rules.
Three permission modes
Allow all writes Allow: /, Disallow all writes Disallow: /, and Custom paths writes one Disallow line for each path you type.
Multiple rule blocks
Add as many blocks as you need. Each one becomes its own group with a blank line between groups, which is the layout crawlers expect.
Sitemap line
If you fill in the Sitemap URL, it is added as a Sitemap: line at the end of the file, where it applies to all crawlers.
Optional crawl delay
A number in seconds is written as Crawl-delay inside every group. Some crawlers honour it; Googlebot does not.
Copy and download
Copy sends the text to your clipboard. Download saves a file named robots.txt, so you do not need to rename anything.
When people build a robots.txt file
- Site owners blocking admin and search pages. A WordPress or custom site can disallow
/wp-admin/or an internal/search?path so crawlers spend time on real content. - Publishers opting out of AI training crawlers. Add a GPTBot or CCBot block set to Disallow all while the
*block stays on Allow all. This asks those bots to skip the whole site. - Developers launching a staging site. Set
*to Disallow all on a staging domain so test pages are not crawled. Remove the rule before going live. - Freelancers handing over a technical checklist. Generate a clean file for a client, then run it through the Robots.txt Tester to confirm key URLs are not blocked by mistake.
Tips before you upload robots.txt
- Do not block CSS or JavaScript folders. Google needs them to render your pages, and blocking them can hurt how pages are understood.
- Disallow all on the
*group blocks the entire site. Use it only for staging or private test sites, and double-check before uploading to production. - A group for a specific bot replaces the
*group for that bot. It does not add to it. If you add a Googlebot block, repeat any paths you also want Googlebot to skip. - The Custom option writes Disallow lines only. If you need an Allow exception, such as
Allow: /wp-admin/admin-ajax.php, add that line by hand after generating. - Leave Crawl Delay empty unless your server struggles. Because the tool writes it into every group, including Googlebot's, it will be ignored there.
- The dropdown list of AI crawlers is fixed. Check each company's own documentation for the bot names it uses today, and add extra groups by hand if needed.
Is it safe? Privacy and limits
The file is built by JavaScript in your browser. Your paths and sitemap URL are not sent anywhere, and the tool does not fetch your site. Copy uses your clipboard and Download creates the file locally. Robots.txt is a public file, so never list secret paths in it. A Disallow line can also point visitors to the very pages you wanted to hide. Use a login or server rules for anything private.
Robots.txt generator FAQ
It is a form that writes a robots.txt file for you. You choose a crawler, say whether it may access everything, nothing or all but certain paths, and the tool outputs the correct lines. This one also adds a Sitemap line and an optional crawl delay. You then upload the finished file to the root of your site. It does not change your website itself, and it does not check how your pages are currently set up. Think of it as a typing aid that gets the syntax and spacing right.
Upload it to the top level of the host it applies to, so it opens at https://yourdomain.com/robots.txt. A file inside a folder, such as /blog/robots.txt, is not read. Each subdomain needs its own file, because blog.example.com and example.com are treated as separate hosts. On WordPress, many SEO plugins create a virtual robots.txt, so check you are not uploading a second copy over it. After uploading, open the address in your browser to confirm the text appears exactly as generated.
Yes. Choose GPTBot, CCBot or anthropic-ai in the User-agent dropdown, set Permission to Disallow all, and generate. Compliant crawlers should then skip your site. Two cautions. Robots.txt is voluntary, so a crawler that ignores it cannot be stopped this way. And the dropdown has a fixed list, so a newer bot name is not covered. Read the crawler documentation from each AI company for current names and add those groups manually if you want to block them.
No. Robots.txt controls crawling, not indexing. If other sites link to a blocked URL, Google can still list the address without a description. To keep a page out of search results, use a noindex meta tag or an HTTP header, and leave the page crawlable so Google can see that instruction. You can create the tag with the Meta Tag Generator. Blocking a page in robots.txt and adding noindex to it at the same time is a common mistake, because the crawler never reads the noindex.
The asterisk is a wildcard that matches any crawler that does not have its own named group in the file. If your file has a group for Googlebot and a group for *, Googlebot follows only the Googlebot group, and all other bots follow the * group. That is why a rule you want for every crawler must be repeated in each specific group. In this tool, choose * (All bots) in the dropdown to write it.
Googlebot ignores the Crawl-delay line. Some other crawlers, including Bing's, may follow it, but support varies. This tool writes your number into every group because it is a single field, so it also lands in Googlebot's group where it has no effect. If Google crawling is overloading your server, the fix is on the server side: improve caching and response time, or return temporary error codes during peaks. For most small sites, leave the field empty.
It is a good idea. A Sitemap line gives every crawler the full address of your XML sitemap, even those you have not submitted to. It is independent of any User-agent group and can appear anywhere in the file; this tool puts it at the end. Use a complete URL starting with https://. If you do not have a sitemap yet, create one with the XML Sitemap Generator and upload it first, so the address in your robots.txt works.
Paste the text into the Robots.txt Tester, enter a few important paths such as your home page, a product page and a blog post, and confirm each is allowed. Then test a path you meant to block. After uploading, open the live file in your browser and check it matches. If something unexpected is blocked, look for a broad rule like Disallow: / or a path that starts the same way as a page you want crawled.