How we check robots.txt files

How the tester decides

For each bot, the tester takes four steps:

1. It finds the bot’s group: a user-agent line that names the bot, and the rules under it.
2. If none does, it tries the bot’s backup name, then the * group, unless the bot ignores it.
3. It compares each rule in the group with the page address, from its first character after the site name.
4. The longest matching rule wins. On a tie, allow wins. If no rule matches, the bot is allowed.

What the tester reads

• user-agent, allow and disallow decide access. sitemap lines never end a group.
• Field names are case-insensitive. Anything after # is a comment.
• A group is one or more user-agent lines, then rules. A user-agent line after a rule starts a new group.
• Rules before the first user-agent line apply to no crawler.
• Only the letters, _ and - at the start of a user-agent value count: Googlebot/2.1 means googlebot.
• Only the first 500 KiB are read: RFC 9309’s minimum, and Google’s limit.
• Lines can end in CR, LF or CRLF. A byte order mark at the start is ignored.
• Spaces and tabs around a value are dropped. A no-break space is kept, as in RFC 9309.

Misspelled fields such as disalow are skipped and flagged. Google’s open-source parser accepts a few of them.

How a crawler picks its group

1. Groups whose user-agent matches the crawler’s token apply. Several matching groups merge into one.
2. Some crawlers list backup tokens: Googlebot-Image falls back to Googlebot. The first token with a group wins.
3. Four Google crawlers ignore *: AdsBot-Google, AdsBot-Google-Mobile, APIs-Google and Mediapartners-Google. So does YandexAdditionalBot, per Yandex.
4. Every other crawler falls back to the * group. A named group and * are never combined.
5. With no matching group and no * group, nothing is blocked.

Apple says Applebot follows Googlebot rules when no group names it. Google does not spell out the order of backup tokens. It says a crawler follows the most specific group that matches, which gives the same result. Yandex accepts YandexAdditional or YandexAdditionalBot, so the tester tries both.

Bing documents no token for four of its fetchers, BingPreview among them. It says a User-agent: * section applies to all bots, so the tester applies * to them.

A fetcher with no documented token or fallback is marked “No token”, and that check comes first. A fetcher whose operator says it ignores robots.txt, or usually does, is marked “Not applied”, with what the rules would have said.

How robots.txt rules match a URL

• Rules match the URL’s path and query from the first character. Case matters: /Fish does not match /fish.
• * matches any run of characters. A final $ means the URL must end there.
• The longest matching rule wins. On a tie between allow and disallow, allow wins.
• An empty disallow: is no rule. /robots.txt itself is always allowed.

Escaped characters, and where Google's parser differs

• Both sides are percent-encoded the same way, so /foo/bar/ツ matches /foo/bar/%E3%83%84.
• Escaped letters and digits such as %62 are decoded first.
• RFC 9309 says crawlers should match a literal * or $ written as %2A or %24. The tester does.

Google’s documentation does not cover these escapes. Its open-source parser matches %62, %2A and %24 as written, so for rules that use them, its result can differ from the tester’s. That parser also reads a line with a space instead of a colon, such as Disallow /private, which the tester skips and flags. And an allow for /dir/index.html also allows /dir/ there, but not in the tester.

Where operators read the file differently

Some operators document a reading that differs from RFC 9309. Where that changes the verdict, the tester marks the crawler “Depends” and shows both.

What each operator says

• Apple’s own example has Applebot obey the * group as well as its own.
• Baidu’s guide says the first matching line decides. It separates groups with blank lines.
• Baidu’s FAQ applies Baiduspider’s rules to Baiduspider-image where no line in the image group matches.
• DuckDuckGo says DuckDuckBot respects WWW::RobotRules, an older Perl parser, so the tester runs a port of it.
• Amazon says Amzn-SearchBot, with no group of its own, follows other search bots’ rules if the file allows them.

For Apple and Baidu, the extra group only adds restrictions. Apple’s example fits two readings: the groups merged, or * applied on its own. Baidu’s guide also says an empty Disallow: allows every URL; other lines are matched the RFC 9309 way, though the guide’s table of examples differs in places.

Baidu’s documents do not settle whether the * group applies to image search when only one of Baiduspider and Baiduspider-image has a group. Nor do they settle a group with no rules, or a user-agent line right after a rule. Where readings disagree, the tester says “either”.

For Amazon, the tester checks the general web-search crawlers the file names: Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot or Baiduspider. If the file shuts them all out of the site, Amazon’s rule does not apply. If they may all crawl the URL and one has no rule disallowing the whole site, Amazon may too. Otherwise the tester says “either”.

DuckDuckBot’s older parser

DuckDuckGo says DuckDuckBot follows an older Perl parser, so the tester runs a port of it.

How the tester reads the file for DuckDuckBot

The port follows WWW::RobotRules and the Perl URI code it uses: Disallow lines only, as plain prefixes of the URL’s escaped path and query. Records end at a blank line, and only the first naming the robot counts.

DuckDuckGo does not say whether the module gets the file as bytes or text, which Perl reads differently for a few characters, so the tester checks both. On October 15, 2026, the port agreed with the real module in five runs of 20,000 distinct cases (how to repeat it).

How we test the tester

The tester’s 56 tests include every example of parsing and matching rules on three Google pages: the robots.txt specification, “Useful robots.txt rules” and “Create and submit a robots.txt file”.

All 56 passed on October 16, 2026. To run them, save robots.mjs as src/lib/robots.mjs and robots.test.mjs as test/robots.test.mjs, then run node --test test/robots.test.mjs. The pages’ examples of which hosts and protocols a file covers are out of scope.

Where the crawler data comes from

The crawler list and the tester share one list, built only from each operator’s own documentation. Each source keeps its URL, the date we read it and a short quote. Where the operator is silent, on a token or an IP list for example, the entry says so.

A script re-fetches every source page and checks that each quote is still there. When one disappears, we reread the page and update the entry and its date. The crawler list shows the last run.

What a robots.txt check cannot tell you

• Whether a crawler obeys the file. The crawler list gives each operator’s word.
• Whether the request gets through. Firewalls, CDN bot settings and rate limits can block an allowed crawler.
• Whether crawlers get the file we read. Errors and redirects change the rules.
• Whether a page gets indexed. A blocked URL can still appear in results, without a snippet.
• Whether a visitor is the real crawler. Anyone can send a Googlebot user agent; verify by IP or reverse DNS.