Free robots.txt checker for AI and search bots
Type your website to see if ChatGPT, Google and other AI can read it.
Result
Example: medium.com
For medium.com's home page, from a copy of its robots.txt saved on October 21, 2026.
- Allowed Google can read this page.
- Bing and Copilot
- Allowed Bing and Copilot can read this page.
- ChatGPT
- Allowed ChatGPT can read this page.
- Claude
- Allowed Claude can read this page.
- Perplexity
- Allowed Perplexity can read this page.
- Gemini
- Allowed Google may use this page for Gemini.
AI training: 4 of 5 blocked. Blocking them is the site's choice, and it does not affect Google or Bing.
Help with these results
Each row is decided by one robots.txt token, named here.
- Googlebot: Allowed. It reads pages for Google Search, including AI Overviews and AI Mode.
- Bing and Copilot
- Bingbot: Allowed. It is Bing's main crawler. Bing says Copilot's search relies on the same crawling and indexing.
- ChatGPT
- OAI-SearchBot: Allowed. It surfaces websites in ChatGPT's search features.
- Claude
- Claude-SearchBot: Allowed. It crawls the web to improve search results for Claude users.
- Perplexity
- PerplexityBot: Allowed. It surfaces and links websites in Perplexity search results.
- Gemini
- Google-Extended: Allowed. It tells Google whether pages it crawls may be used to train Gemini and to ground answers in Gemini Apps. It is not a crawler of its own.
See all 65 bots
Search engine bots: 14 allowed, 2 unclear.
| Bot | Company | Result | Why |
|---|---|---|---|
Amzn-SearchBot | Amazon | Allowed | No rule covers this page. |
Applebot | Apple | Allowed | No rule covers this page. |
Baiduspider | Baidu | Unclear | RFC 9309: allowed. No rule covers this page. Baidu's documentation: either. Why they differ |
Baiduspider-image | Baidu | Unclear | RFC 9309: allowed. No rule covers this page. Baidu's documentation: either. Why they differ |
Bingbot | Microsoft | Allowed | No rule covers this page. |
| Brave Search crawler | Brave | Allowed | No rule covers this page. |
Diffbot | Diffbot | Allowed | No rule covers this page. Diffbot's crawls follow robots.txt by default, including disallow and crawl-delay, but can ignore it in specific cases, typically under an agreement with the crawled site. |
DuckDuckBot | DuckDuckGo | Allowed | No rule covers this page. |
Googlebot | Allowed | No rule covers this page. | |
Googlebot-Image | Allowed | No rule covers this page. | |
Googlebot-News | Allowed | No rule covers this page. | |
Googlebot-Video | Allowed | No rule covers this page. | |
Storebot-Google | Allowed | No rule covers this page. | |
YandexBot | Yandex | Allowed | No rule covers this page. |
YandexImages | Yandex | Allowed | No rule covers this page. |
YandexVideo | Yandex | Allowed | No rule covers this page. |
AI search bots: 6 allowed.
| Bot | Company | Result | Why |
|---|---|---|---|
Claude-SearchBot | Anthropic | Allowed | No rule covers this page. |
DuckAssistBot | DuckDuckGo | Allowed | No rule covers this page. |
meta-webindexer | Meta | Allowed | No rule covers this page. |
MistralAI-Index | Mistral AI | Allowed | No rule covers this page. |
OAI-SearchBot | OpenAI | Allowed | No rule covers this page. |
PerplexityBot | Perplexity | Allowed | No rule covers this page. |
AI training bots: 1 allowed, 4 blocked.
| Bot | Company | Result | Why |
|---|---|---|---|
Amazonbot | Amazon | Blocked | Line 30: Disallow: / |
ClaudeBot | Anthropic | Blocked | Line 30: Disallow: / |
GPTBot | OpenAI | Blocked | Line 30: Disallow: / |
meta-externalagent | Meta | Blocked | Line 30: Disallow: / |
MistralAI-Training | Mistral AI | Allowed | No rule covers this page. |
Bots that open a page when a person asks: 5 allowed, 11 not decided by robots.txt.
| Bot | Company | Result | Why |
|---|---|---|---|
Amzn-User | Amazon | Allowed | No rule covers this page. Amazon says that because its actions can be user-initiated, Amzn-User may not follow all robots.txt directives. |
ChatGPT-User | OpenAI | Not decided by robots.txt | OpenAI documents no robots.txt token for it. OpenAI says robots.txt rules may not apply because these actions are user-initiated, and names only OAI-SearchBot and GPTBot as robots.txt tags. ChatGPT-User is not used to decide Search inclusion. |
Claude-User | Anthropic | Allowed | No rule covers this page. |
Diffbot-User | Diffbot | Allowed | No rule covers this page. Diffbot says its crawls follow robots.txt by default but can ignore it under an agreement with the crawled site. It lists Diffbot-User among the user agents to name in robots.txt. |
FeedFetcher-Google | Not decided by robots.txt | Google documents no robots.txt token for it. Google says Feedfetcher ignores robots.txt because its requests come from explicit user action; it suggests serving a 404, 410 or other error to the Feedfetcher-Google user agent instead. | |
Google-Agent | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
Google-CWS | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
Google-GeminiNotebook | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
Google-Pinpoint | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
Google-Read-Aloud | Not decided by robots.txt | Google documents no robots.txt token for it. Google says you cannot opt out with robots.txt because Read Aloud is user-initiated; it offers the nopagereadaloud meta tag instead. The older agent name google-speakr is deprecated. | |
Google-Site-Verification | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
GoogleMessages | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
GoogleProducer | Not decided by robots.txt | Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string. | |
meta-externalfetcher | Meta | Allowed | No rule covers this page. Meta says it may bypass robots.txt because it performs fetches requested by the user. |
MistralAI-User | Mistral AI | Allowed | No rule covers this page. |
Perplexity-User | Perplexity | Not decided by robots.txt | Perplexity says this fetcher generally ignores robots.txt because a user requested the fetch, although the same page lists Perplexity-User among the robots.txt tags webmasters can use. The rules would allow it. |
AI opt-outs: 1 opted out, 2 allowed.
| Bot | Company | Result | Why |
|---|---|---|---|
Applebot-Extended | Apple | Opted out | Line 30: Disallow: / |
Google-Extended | Allowed | No rule covers this page. | |
YandexAdditionalBot | Yandex | Allowed | No group names it, and it skips the * group. |
Other bots: 13 allowed, 3 blocked, 3 not decided by robots.txt.
| Bot | Company | Result | Why |
|---|---|---|---|
AdIdxBot | Microsoft | Allowed | No rule covers this page. |
AdsBot-Google | Allowed | No group names it, and it skips the * group. Ignores the global user agent (*) with the ad publisher's permission; rules addressed to AdsBot-Google apply. | |
AdsBot-Google-Mobile | Allowed | No group names it, and it skips the * group. Ignores the global user agent (*); rules addressed to AdsBot-Google-Mobile apply. | |
APIs-Google | Allowed | No group names it, and it skips the * group. Ignores the global user agent (*); rules addressed to APIs-Google apply. | |
BingPreview | Microsoft | Allowed | No rule covers this page. |
BingVideoPreview | Microsoft | Allowed | No rule covers this page. |
CCBot | Common Crawl | Allowed | No rule covers this page. |
facebookexternalhit | Meta | Allowed | No rule covers this page. Meta says it might bypass robots.txt when performing security or integrity checks. |
Google-CloudVertexBot | Allowed | No rule covers this page. | |
Google-InspectionTool | Allowed | No rule covers this page. | |
Google-Safety | Not decided by robots.txt | Google documents no robots.txt token for it. Google says this user agent ignores robots.txt rules. | |
GoogleOther | Blocked | Line 30: Disallow: / | |
GoogleOther-Image | Blocked | Line 30: Disallow: / | |
GoogleOther-Video | Blocked | Line 30: Disallow: / | |
iTMS | Apple | Not decided by robots.txt | Apple says the iTMS user agent does not follow robots.txt because it is not a general search crawler. Apple documents no robots.txt token for it. |
Mediapartners-Google | Allowed | No group names it, and it skips the * group. Ignores the global user agent (*); rules addressed to Mediapartners-Google apply. | |
meta-externalads | Meta | Allowed | No rule covers this page. |
MicrosoftPreview | Microsoft | Allowed | No rule covers this page. |
OAI-AdsBot | OpenAI | Not decided by robots.txt | OpenAI documents no robots.txt token or rules for OAI-AdsBot. It says the bot visits only pages submitted as ads. |
What each bot does: see the crawler list.
Still blocked?
A bot that your robots.txt allows can still be turned away. These fixes cover common causes.