robots.txt tester for AI and search bots
Paste your robots.txt file and a page address to see which AI and search bots may visit that page, and the line that decides, if any.
Results
To unblock a bot, change the line shown under "Decided by", or add a longer Allow line for this page to the same group. Then test again. If the row then says "Depends", read what the operator's documentation says under "Decided by".
No crawlers match these filters.
What this test covers
- Rules
- Covered: what your file allows each bot on this page, or why the tester cannot tell.
- Response
- How the site answers a request for its robots.txt is checked only when this page's address ends in ?site= and a site name, and only as our checker got it. See robots.txt returns errors, redirects or HTML.
- Identity
- Whether a visitor is the real bot is not tested here. See how to check a visitor is the real bot.
What the robots.txt tester shows
The example already in the form gives these results for 8 of the bots and tokens:
User-agent: *
Allow: /
User-agent: GPTBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /blog/| Bot | Result | Decided by |
|---|---|---|
GooglebotSearch | Allowed | allow: /line 2, group * |
BingbotSearch | Allowed | allow: /line 2, group * |
OAI-SearchBotAI search | Allowed | allow: /line 2, group * |
PerplexityBotAI search | Allowed | allow: /line 2, group * |
GPTBotAI training | Blocked | disallow: /blog/line 7, group GPTBot |
CCBotOther | Blocked | disallow: /blog/line 7, group CCBot |
Google-ExtendedControl token | Blocked | disallow: /blog/line 7, group Google-Extended |
ChatGPT-UserUser-triggered | No token | No documented robots.txt token |
OpenAI says robots.txt rules "may not apply" to ChatGPT-User, which fetches pages when a person asks.
How to read your file by hand
- Find every group whose
User-agentline names the bot (capital letters do not matter). Treat them as one group. - If none does, try the bot's backup name, then the
*group. - Compare each rule with the part of the page address after the site name, from its first character.
- In a rule, a
*matches anything, and a$at the end means the address must end there. - The longest matching rule wins. If an
allowand adisallowrule are the same length,allowwins. If no rule matches, the bot is allowed.
The tester checks the 65 bots and tokens in the crawler list. The method covers encoding, size limits and what robots.txt cannot tell you.
Help with the results
What "Not applied" means
The operator says the crawler does not follow robots.txt, or usually does not, often because a person asked it to fetch the page. The tester still shows what the rules would say.
What "No token" means
The operator documents no robots.txt token for that fetcher, so there is no group to look up. Google's user-triggered fetchers are an example.
What "Depends" means
The operator's own documentation reads robots.txt differently from RFC 9309, and here the two readings disagree. The result shows both. The method lists these cases.
Why a bot is allowed when your * group blocks everything
A group naming it may allow the URL, or it may ignore the * group, as Google's AdsBot does. The "Decided by" column shows which.
What happens to your file
What you paste or type here stays in your browser. Our server fetches a site's /robots.txt only when this page's address ends in ?site= and a site name. The home page's check sends you there when a file is too long for a link. Our server fetches the file once and does not save it. A share link carries the file after the # in the address, and browsers do not send that part to servers.