What is a robots.txt tester?
One wrong line in robots.txt can hide a whole section of your site from search engines. A Disallow: / left over from staging is the classic case. A robots.txt tester lets you check the effect of your rules before they go live.
With this robots.txt tester you paste the file into the left box. The right side shows the parsed rules as you type, with a short validation list and any Sitemap: lines it found. Below that, enter a path (not a full URL), pick a user agent button and click Test. The result says ALLOWED or BLOCKED, shows the rule that matched, and notes which group applied.
It does not fetch your live robots.txt, and it does not test full URLs on a domain, only the path. It also uses a simplified matcher: the first group for the exact agent name wins, then the wildcard group, and the longest matching rule decides. Use it as a fast check and confirm important decisions with Google Search Console.
How to test a URL against your robots.txt
- Paste your file into Paste Your robots.txt Content, or click Load Example to see a sample.
- Read Parsed Rules and Validation on the right. They update as you type.
- Enter a path in URL to Test (path only), for example /blog/post-name or /private/report.pdf.
- Choose a user agent: All (*), Googlebot, Bingbot or Twitterbot.
- Click Test. Read the verdict and the matching rule shown under it.
- Check Detected Sitemaps at the bottom, and use Clear to start over with a new file.
What this robots.txt tester checks
Live parsing
Rules are colour-coded as you type: user-agent lines, Disallow, Allow, comments, sitemaps and unknown lines.
Allowed or blocked verdict
The result shows the user agent, the path tested, the rule that matched and which group it came from.
Four user agent choices
Test as All (*), Googlebot, Bingbot or Twitterbot with one click.
Validation messages
The tool warns about a Disallow: / line, lines without a colon, and Crawl-delay, which Googlebot ignores.
Wildcard and end-of-URL support
Rules with * are treated as wildcards. A longer matching rule beats a shorter one, and Allow wins a tie.
Sitemap detection
Any Sitemap: lines are listed as links at the bottom of the page.
Who uses it, and for what
- WordPress site owners. Check that /wp-admin/ is blocked while /wp-admin/admin-ajax.php stays open, which is what the example file shows.
- SEOs auditing a migration. Paste the new site's file and test ten important paths before launch day.
- Developers editing a file. After using the Robots.txt Generator, paste the output here and confirm that nothing important is blocked.
- Content teams chasing a missing page. If a page does not appear in search, test its path to rule robots.txt in or out, then look at the Meta Tag Analyzer for a noindex tag.
Tips and known limits
- Enter the path only, starting with a slash. Do not paste the full https:// address.
- Paths are matched case-sensitively, like real crawlers do, so /Admin/ and /admin/ are different.
- The matcher turns each rule into a pattern without escaping other characters. A rule containing ? or + may not match as expected, for example a rule like /?s=, so verify query-string rules in Search Console.
- A bot with its own group ignores the wildcard group. If Bingbot has only a Crawl-delay line, the * rules do not apply to it in this tool.
- Only exact user-agent names are matched. A group for Googlebot-Image will not be used when you test Googlebot.
- Testing is not the same as indexing. A page that is allowed can still be left out of results for other reasons.
Is it safe? Privacy and limits
Parsing and testing happen in your browser. The text you paste is not uploaded or stored by the tool, and it does not fetch your real robots.txt or any URL. The result only reflects the pasted text and a simplified reading of the rules, so it is a guide. For decisions that affect indexing, verify with the robots.txt report in Google Search Console.
Robots.txt tester FAQ
It reads the rules in a robots.txt file and tells you what would happen to a particular URL path for a particular crawler. You get an allowed or blocked result and the rule responsible. This tester also flags simple problems, such as a Disallow: / line or lines it cannot read. It is useful for checking edits before you upload them, or for finding out why a page might not be crawled. It does not fix the file for you.
Paste your robots.txt into the main box, type the path of the page in the test field, choose a user agent and click Test. For https://example.com/blog/post, enter /blog/post. If a Disallow rule matches and no longer Allow rule overrides it, the result says BLOCKED and shows the rule. If no rule matches, the path is allowed by default. Test several important pages, not just one.
No. The tool does not fetch anything from the web, so you need to copy your file's content and paste it in. Open yourdomain.com/robots.txt in a browser, select all and copy. This also means you can test changes that are not published yet. If you need to confirm what Google actually sees on your live site, use the robots.txt report in Google Search Console.
It first picks a group: the first group that names the chosen user agent exactly, otherwise the first * group. Inside the group it finds every Allow and Disallow rule whose path matches the start of the URL path, and the longest rule wins. If an Allow and a Disallow are the same length, Allow wins. An empty Disallow means nothing is blocked. This follows the common behaviour, but real crawlers differ in small details.
Googlebot does not support the Crawl-delay directive, so the line has no effect on Google. Some other crawlers, such as Bingbot, may follow it. The warning is there to avoid a false sense of control. If you need to slow Google down, use the crawl settings available in Search Console, or fix the server speed. Keeping the line for other bots is fine.
No. Allowed only means robots.txt does not stop the crawler from requesting the URL. Pages can still be left out for other reasons: a noindex tag, a canonical pointing elsewhere, low quality, soft 404 errors or not being discovered yet. Likewise, Blocked means the crawler will not fetch the page, but the bare URL can still be listed if others link to it. Use the right tool for each question.
The buttons are All (*), Googlebot, Bingbot and Twitterbot. The tool matches the name you pick against the User-agent lines in your file, ignoring case. Other crawlers, such as GPTBot or Googlebot-Image, have no button here, but if you want to check an unusual bot you can edit a copy of your file so the group you care about uses one of the four names. Remember that variants are not matched automatically.
Yes, it is free with no account, and it does not limit how many files or paths you test. After testing, fix any problems and run the file again. You can build a fresh file with the Robots.txt Generator if yours is messy. Always keep a backup of the working file before uploading changes, in case a rule blocks something important.