AI and search crawler list

Crawlers are bots that read websites. 65 crawlers and robots.txt tokens: what each does, whether its operator says it follows robots.txt, and how to verify it where documented.

From each operator's own documentation. "Follows robots.txt" is the operator's word, not our test. We last re-checked the source quotes on October 22, 2026: 166 of 166 found (report, script). A source with no date of its own was read on the Checked date in its row's details. How we keep it current.

Search engine bots

These read your pages so people can find them in search engines like Google and Bing.

Crawler	What it does	Follows robots.txt	Verify by
`Amzn-SearchBot`Amazon	Improves search experiences in Amazon products and services such as Alexa; Amazon says it does not crawl for generative AI model training. Details for Amzn-SearchBot User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36` robots.txt If robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the rules given to other search bots (Amazon names no specific token). Verification Amazon publishes the IP list as JSON inside an HTML page; no reverse DNS method is documented on the current page. Sources About AmazonBot, read October 14, 2026 Checked October 15, 2026	YesNote 1	IP list
`Applebot`AppleThen `Googlebot`	Apple's web crawler for search in Spotlight, Siri and Safari; Apple says its data may also be used to train Apple foundation models unless Applebot-Extended is disallowed. Details for Applebot User agent `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)` robots.txt In general search crawls, Applebot respects robots.txt directives aimed at it. If robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot rules. It does not follow Crawl-delay. The UA shown is Apple's desktop example. Verification Reverse DNS in *.applebot.apple.com, confirmed with a forward lookup, or match the IP against Apple's published CIDR list. Sources About Applebot Checked October 15, 2026	YesNote 2	IP list Reverse DNS: `applebot.apple.com`
`Baiduspider`Baidu	Baidu's search engine crawler, which visits pages to build the index for Baidu web and mobile search. Details for Baiduspider User agent `Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html）` robots.txt Baidu says Baiduspider follows the robots protocol and that a Baiduspider disallow blocks all Baidu products. Baiduspider-cpro and Baiduspider-ads do not follow robots.txt. The page is in Chinese. Verification Reverse DNS hostname must end in .baidu.com or .baidu.jp, confirmed with a forward DNS lookup. Baidu publishes no IP list on this page. Sources Baiduspider常见问题解答 (Baiduspider FAQ) Robots文件设置 (robots.txt settings) Checked October 16, 2026	YesNote 3	Reverse DNS: `baidu.com`, `baidu.jp`
`Baiduspider-image`BaiduThen `Baiduspider`	User agent for Baidu image search. Details for Baiduspider-image User agent Not documented robots.txt Baidu says each product's user agent can get its own rules, and that disallowing Baiduspider blocks all Baidu products. Verification Reverse DNS hostname must end in .baidu.com or .baidu.jp, confirmed with a forward DNS lookup. Baidu publishes no IP list on this page. Sources Baiduspider常见问题解答 (Baiduspider FAQ) Robots文件设置 (robots.txt settings) Checked October 16, 2026	YesNote 4	Reverse DNS: `baidu.com`, `baidu.jp`
Brave Search crawlerBraveNo documented robots.txt tokenFollows `Googlebot`	Crawls pages for Brave Search's own index. It does not advertise a user agent of its own. Details for Brave Search crawler User agent Not documented robots.txt Brave says its crawler will not crawl a page or domain that Googlebot cannot crawl, so the rules Googlebot gets apply to it. Verification No verification method documented; Brave says the crawler does not advertise a differentiated user agent. Sources Brave Search Crawler Checked October 15, 2026	Yes	Not documented
`Diffbot`Diffbot	General web crawling for a search engine, so sites can be found and cited from the Diffbot Knowledge Graph and web search services; Diffbot says it is not used for AI training. Details for Diffbot User agent Not documented Sources Does Diffbot respect robots.txt? Checked October 14, 2026	PartlyNote 5	Not documented
`DuckDuckBot`DuckDuckGo	DuckDuckGo's web crawler for improving its search results. Details for DuckDuckBot User agent `DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)` robots.txt DuckDuckGo says the crawler respects WWW::RobotRules. Verification DuckDuckGo lists IP addresses on its page and in a JSON file. DuckDuckBot and DuckAssistBot share the same JSON file, which has more addresses than either page. No reverse DNS method is documented. Sources Is DuckDuckBot related to DuckDuckGo?, read October 14, 2026 WWW::RobotRules Checked October 15, 2026	YesNote 6	IP list
`Googlebot`Google	Google's main crawler for Google Search, including AI Overviews and AI Mode, with a smartphone and a desktop type that share one robots.txt token. Details for Googlebot User agent Smartphone: `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` Desktop: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36` Rarely: `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` Rarely: `Googlebot/2.1 (+http://www.google.com/bot.html)` robots.txt Google's common crawlers always obey robots.txt for automatic crawls. Googlebot Smartphone and Googlebot Desktop cannot be targeted separately in robots.txt. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers, read October 14, 2026 What Is Googlebot, read October 14, 2026 AI features and your website Checked October 21, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Googlebot-Image`GoogleThen `Googlebot`	Crawls images for Google Images, Discover, Video mode and Search features that show images, logos and favicons. Details for Googlebot-Image User agent `Googlebot-Image/1.0` robots.txt Common crawler; obeys robots.txt for automatic crawls. Google lists both Googlebot-Image and Googlebot as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers Checked October 15, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Googlebot-News`GoogleThen `Googlebot`	Robots.txt token for the Google News product; Google says it has no separate user agent string and crawling uses the various Googlebot strings. Details for Googlebot-News User agent None of its own. Google crawls for Google News with Googlebot's user agent strings. robots.txt Common crawler token; Google lists both Googlebot-News and Googlebot as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers, read October 14, 2026 Checked October 15, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Googlebot-Video`GoogleThen `Googlebot`	Crawls video content for video-related Google Search features and other products that depend on videos. Details for Googlebot-Video User agent `Googlebot-Video/1.0` robots.txt Common crawler; obeys robots.txt for automatic crawls. Google lists both Googlebot-Video and Googlebot as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers Checked October 14, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Storebot-Google`Google	Crawler for Google Shopping surfaces, including the Shopping tab in Google Search. Details for Storebot-Google User agent Desktop: `Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36` Mobile: `Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36` robots.txt Common crawler; obeys robots.txt for automatic crawls. Google lists only Storebot-Google as its token. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers, read October 14, 2026 Checked October 15, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Bingbot`Microsoft	Bing's standard crawler, which handles most of Bing's crawling. Bing says its Copilot search experiences rely on the same crawling and indexing as its search. Details for Bingbot User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36` robots.txt Bing says robots.txt controls its crawlers. When Bingbot finds a group for itself it ignores the generic (*) group. Verification Reverse DNS lookup should end in search.msn.com, then a forward lookup must return the same IP; or match the IP against bingbot.json (Bing says to refresh it daily). Bing also offers a Verify Bingbot tool. Sources Overview of Bing crawlers (user agents), read October 14, 2026 How to create a robots.txt file, read October 14, 2026 Verifying Bingbot, read October 14, 2026 Bing Webmaster Guidelines Checked October 22, 2026	Yes	IP list Reverse DNS: `search.msn.com`
`YandexBot`YandexThen `Yandex`	Yandex's main indexing robot. Details for YandexBot User agent `Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)` robots.txt Yandex's table says the main indexing robot takes the general robots.txt rules into account. A User-agent: Yandex group applies to Yandex robots, and when one is present the * group is ignored. Verification Reverse DNS hostname must end in yandex.ru, yandex.net or yandex.com, confirmed with a forward lookup. Yandex does not disclose its IP list. Sources The User-agent directive How to check that a robot belongs to Yandex Checked October 14, 2026	Yes	Reverse DNS: `yandex.ru`, `yandex.net`, `yandex.com`
`YandexImages`YandexThen `Yandex`	Indexes images for Yandex Images. Details for YandexImages User agent `Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)` robots.txt Yandex's table says this robot takes the general robots.txt rules into account. Verification Reverse DNS hostname must end in yandex.ru, yandex.net or yandex.com, confirmed with a forward lookup. Yandex does not disclose its IP list. Sources How to check that a robot belongs to Yandex The User-agent directive Checked October 14, 2026	Yes	Reverse DNS: `yandex.ru`, `yandex.net`, `yandex.com`
`YandexVideo`YandexThen `Yandex`	Indexes videos for Yandex video search. Details for YandexVideo User agent `Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)` robots.txt Yandex's table says this robot takes the general robots.txt rules into account. Verification Reverse DNS hostname must end in yandex.ru, yandex.net or yandex.com, confirmed with a forward lookup. Yandex does not disclose its IP list. Sources How to check that a robot belongs to Yandex The User-agent directive Checked October 14, 2026	Yes	Reverse DNS: `yandex.ru`, `yandex.net`, `yandex.com`

Notes on robots.txt: search engine bots
Crawler	Note
1`Amzn-SearchBot`	Amazon says that when robots.txt does not mention it but allows other search bots, it follows their rules. Amazon does not say which bots.
2`Applebot`	Apple's example has Applebot also skip paths the * group disallows. It fits two readings: the groups merged, or the * group applied on its own.
3`Baiduspider`	Baidu's robots.txt guide says the first Allow or Disallow line that matches decides, in the order written, where RFC 9309 takes the longest. It says an empty Disallow allows every URL, and it separates groups with blank lines. The tester matches each line the RFC 9309 way. Where a group has no rules, or a user-agent line comes right after a rule, Baidu's documents do not settle which lines apply.
4`Baiduspider-image`	Baidu's guide says the first matching line decides and an empty Disallow allows every URL. It separates groups with blank lines. Baidu's example also applies Baiduspider's rules to image search where the image group has no matching line, though the guide says a crawler with its own group follows only that group. Where only one of Baiduspider and Baiduspider-image has a group, Baidu's documents do not settle whether the * group applies. Nor do they settle a group with no rules, or a user-agent line right after a rule.
5`Diffbot`	Diffbot's crawls follow robots.txt by default, including disallow and crawl-delay, but can ignore it in specific cases, typically under an agreement with the crawled site.
6`DuckDuckBot`	DuckDuckGo says DuckDuckBot respects WWW::RobotRules, an older parser: Disallow lines only, as plain prefixes, and records end at a blank line. A few characters, such as a no-break space, read differently when it gets the file as text rather than bytes.

AI search bots

These crawl or index pages for AI answers and AI search.

Crawler	What it does	Follows robots.txt	Verify by
`Claude-SearchBot`Anthropic	Crawls the web to improve the quality and relevance of search results for Claude users. Details for Claude-SearchBot User agent Not documented robots.txt Anthropic says its bots honor robots.txt. Disabling Claude-SearchBot stops indexing for search optimization and may reduce visibility in user search results. Verification Anthropic links one IP list for its bots and says a source IP on that list means the crawler is from Anthropic. It does not split the list by bot or document reverse DNS. Sources Does Anthropic crawl data from the web, and how can site owners block the crawler? Checked October 14, 2026	Yes	IP list
`DuckAssistBot`DuckDuckGo	Crawls pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources; DuckDuckGo says the data is not used to train AI models. Details for DuckAssistBot User agent `DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)` robots.txt Publishers can opt out by disallowing DuckAssistBot in robots.txt; the change takes effect after 72 hours and does not affect organic rankings. Verification DuckDuckGo lists IP addresses on its page and in a JSON file. DuckDuckBot and DuckAssistBot share the same JSON file, which has more addresses than either page. No reverse DNS method is documented. Sources Is DuckAssistBot related to DuckDuckGo?, read October 14, 2026 Checked October 15, 2026	Yes	IP list
`meta-webindexer`Meta	Crawls the web to improve Meta AI search result quality so Meta AI can cite and link content. Details for meta-webindexer User agent `meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)` robots.txt Meta says to block its crawlers with a robots.txt disallow and that allowing Meta-WebIndexer helps Meta AI cite and link content. Verification Meta publishes no IP list or reverse DNS method for this crawler. Sources Meta Web Crawlers, read October 14, 2026 Checked October 15, 2026	Yes	Not documented
`MistralAI-Index`Mistral AI	Crawls the web to index content for Mistral search, which helps answer questions in Vibe; Mistral says the content is not used for generative AI training. Details for MistralAI-Index User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)` robots.txt Listed among the robots.txt tags Mistral offers webmasters; no crawler-specific robots.txt wording. Verification Mistral publishes IP addresses; no reverse DNS method is documented. Sources Mistral crawlers, read October 14, 2026 Checked October 15, 2026	Yes	IP list
`OAI-SearchBot`OpenAI	Surfaces websites in ChatGPT's search features. Details for OAI-SearchBot User agent `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot` robots.txt Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers but can still appear as navigational links; changes can take about 24 hours. The documented UA is an example; the version may change. Verification OpenAI publishes IP addresses for OAI-SearchBot; no reverse DNS method is documented. Sources Overview of OpenAI Crawlers Checked October 14, 2026	Yes	IP list
`PerplexityBot`Perplexity	Surfaces and links websites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models. Details for PerplexityBot User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` robots.txt Perplexity says webmasters can use robots.txt tags to manage access; changes may take up to 24 hours. Verification Perplexity publishes IP addresses and recommends combining user agent matching with IP verification. Sources Perplexity Crawlers Checked October 14, 2026	Yes	IP list

AI training bots

These collect pages that may be used to train AI models.

Crawler	What it does	Follows robots.txt	Verify by
`Amazonbot`Amazon	Used to improve Amazon products and services; Amazon says its data may be used to train Amazon AI models, so the purpose is mixed. Details for Amazonbot User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36` robots.txt Amazon says automated crawling respects robots.txt allow/disallow rules but not Crawl-delay; robots.txt may be cached for up to 30 days. Verification Amazon publishes the IP list as JSON inside an HTML page; no reverse DNS method is documented on the current page. Sources About AmazonBot Checked October 14, 2026	Yes	IP list
`ClaudeBot`Anthropic	Collects web content that could contribute to training Anthropic's generative AI models. Details for ClaudeBot User agent Not documented robots.txt Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay extension. Restricting ClaudeBot signals that future materials should be excluded from training data. Verification Anthropic links one IP list for its bots and says a source IP on that list means the crawler is from Anthropic. It does not split the list by bot or document reverse DNS. Sources Does Anthropic crawl data from the web, and how can site owners block the crawler? Checked October 14, 2026	Yes	IP list
`meta-externalagent`Meta	Crawls the web for uses such as training foundation AI models or improving products by indexing content directly (Meta names both uses). Details for meta-externalagent User agent `meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)` robots.txt Meta says to block its crawlers with a robots.txt disallow; robots.txt may be cached for up to 24 hours. Verification Meta publishes no IP list or reverse DNS method for this crawler. Sources Meta Web Crawlers, read October 14, 2026 Checked October 15, 2026	Yes	Not documented
`MistralAI-Training`Mistral AI	Crawls web content to build datasets for training Mistral generative AI models. Details for MistralAI-Training User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)` robots.txt Mistral says webmasters can disallow this user agent in robots.txt. Verification Mistral's page lists no IP addresses for this crawler. Sources Mistral crawlers Checked October 14, 2026	Yes	Not documented
`GPTBot`OpenAI	Crawls content that may be used to train OpenAI's generative AI foundation models. Details for GPTBot User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot` robots.txt Disallowing GPTBot signals that a site's content should not be used to train OpenAI's foundation models. The documented UA is an example; the version number may change. Verification OpenAI publishes IP addresses for GPTBot; no reverse DNS method is documented. Sources Overview of OpenAI Crawlers Checked October 14, 2026	Yes	IP list

Bots that open a page when a person asks

These fetch a page because a person asked.

Crawler	What it does	Follows robots.txt	Verify by
`Amzn-User`Amazon	Fetches live web information on a user's behalf, for example to answer Alexa queries; Amazon says it does not crawl for generative AI model training. Details for Amzn-User User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36` Verification Amazon publishes the IP list as JSON inside an HTML page; no reverse DNS method is documented on the current page. Sources About AmazonBot Checked October 14, 2026	PartlyNote 1	IP list
`Claude-User`Anthropic	Accesses websites when individuals ask Claude questions. Details for Claude-User User agent Not documented robots.txt Anthropic says its bots honor robots.txt; disabling Claude-User prevents retrieval of a site's content in response to a user query. Verification Anthropic links one IP list for its bots and says a source IP on that list means the crawler is from Anthropic. It does not split the list by bot or document reverse DNS. Sources Does Anthropic crawl data from the web, and how can site owners block the crawler? Checked October 14, 2026	Yes	IP list
`Diffbot-User`Diffbot	Requests made on behalf of a human user browsing a URL with Diffbot software. Details for Diffbot-User User agent Not documented Sources Does Diffbot respect robots.txt? Checked October 15, 2026	PartlyNote 2	Not documented
`FeedFetcher-Google`GoogleNo documented robots.txt token	Crawls RSS or Atom feeds for Google News and WebSub at the request of users of apps or services. Details for FeedFetcher-Google User agent `FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)` Verification Google's Feedfetcher page says its IP addresses are in user-triggered-fetchers-google.json. Sources Google User-Triggered Fetchers, read October 14, 2026 Google Feedfetcher, read October 14, 2026 Checked October 15, 2026	NoNote 3	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`Google-Agent`GoogleNo documented robots.txt token	Used by agents hosted on Google infrastructure to browse the web and take actions when a user asks. Details for Google-Agent User agent Mobile: `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)` Desktop: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36` Verification Google says this fetcher uses IP ranges from user-triggered-agents.json. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 4	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`Google-CWS`GoogleNo documented robots.txt token	Chrome Web Store fetcher that requests URLs developers provide in the metadata of their Chrome extensions and themes. Details for Google-CWS User agent `Mozilla/5.0 (compatible; Google-CWS)` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 5	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`Google-GeminiNotebook`GoogleNo documented robots.txt token	Fetches individual URLs that Gemini Notebook users add as sources for their projects. Google lists its former name, Google-NotebookLM, as supported until August 2026. Details for Google-GeminiNotebook User agent Mobile: `Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Mobile Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)` Desktop: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 6	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`Google-Pinpoint`GoogleNo documented robots.txt token	Fetches individual URLs that Pinpoint users name as sources for their document collections. Details for Google-Pinpoint User agent `Google-Pinpoint` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 7	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`Google-Read-Aloud`GoogleNo documented robots.txt token	Fetches and reads web pages aloud with text-to-speech when a user asks. Details for Google-Read-Aloud User agent Mobile: `Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)` Desktop: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Google Read Aloud User Agent, read October 14, 2026 Checked October 15, 2026	NoNote 8	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`Google-Site-Verification`GoogleNo documented robots.txt token	Fetches Search Console verification tokens on a user's request. Details for Google-Site-Verification User agent `Mozilla/5.0 (compatible; Google-Site-Verification/1.0)` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 9	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`GoogleMessages`GoogleNo documented robots.txt token	Generates link previews for URLs sent in Google Messages chats. Details for GoogleMessages User agent `GoogleMessages` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 10	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`GoogleProducer`GoogleNo documented robots.txt token	Google Publisher Center fetcher that processes feeds publishers supply for Google News landing pages. Details for GoogleProducer User agent `GoogleProducer; (+https://developers.google.com/search/docs/crawling-indexing/google-producer)` Verification Google says fetchers it controls use IPs in user-triggered-fetchers-google.json (google-proxy-***.google.com); IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com. The docs do not map each fetcher to one file. Sources Google User-Triggered Fetchers, read October 14, 2026 Checked October 15, 2026	Usually notNote 11	IP list Reverse DNS: `google.com`, `googleusercontent.com`
`meta-externalfetcher`Meta	Fetches individual links at a user's request and supports product functions such as agentic AI tasks. Details for meta-externalfetcher User agent `meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)` Verification Meta publishes no IP list or reverse DNS method for this crawler. Sources Meta Web Crawlers, read October 14, 2026 Checked October 15, 2026	PartlyNote 12	Not documented
`MistralAI-User`Mistral AI	Visits web pages for user actions in Vibe, Mistral's assistant, and may link the source in its answer. Details for MistralAI-User User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)` robots.txt Mistral describes MistralAI-User as a robots.txt tag that governs which sites user requests can reach; it does not say the fetcher ignores robots.txt. Verification Mistral publishes IP addresses; no reverse DNS method is documented. Sources Mistral crawlers Checked October 14, 2026	Yes	IP list
`ChatGPT-User`OpenAINo documented robots.txt token	Visits web pages for certain user actions in ChatGPT and Custom GPTs; OpenAI says it is not used for automatic crawling. Details for ChatGPT-User User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot` Verification OpenAI publishes IP addresses for ChatGPT-User; no reverse DNS method is documented. Sources Overview of OpenAI Crawlers Checked October 15, 2026	UnknownNote 13	IP list
`Perplexity-User`Perplexity	Visits web pages when users ask Perplexity a question, to help answer and link the page. Details for Perplexity-User User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` Verification Perplexity publishes IP addresses and recommends combining user agent matching with IP verification. Sources Perplexity Crawlers, read October 14, 2026 Checked October 15, 2026	Usually notNote 14	IP list

Notes on robots.txt: bots that open a page when a person asks
Crawler	Note
1`Amzn-User`	Amazon says that because its actions can be user-initiated, Amzn-User may not follow all robots.txt directives.
2`Diffbot-User`	Diffbot says its crawls follow robots.txt by default but can ignore it under an agreement with the crawled site. It lists Diffbot-User among the user agents to name in robots.txt.
3`FeedFetcher-Google`	Google says Feedfetcher ignores robots.txt because its requests come from explicit user action; it suggests serving a 404, 410 or other error to the Feedfetcher-Google user agent instead.
4`Google-Agent`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
5`Google-CWS`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
6`Google-GeminiNotebook`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
7`Google-Pinpoint`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
8`Google-Read-Aloud`	Google says you cannot opt out with robots.txt because Read Aloud is user-initiated; it offers the nopagereadaloud meta tag instead. The older agent name google-speakr is deprecated.
9`Google-Site-Verification`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
10`GoogleMessages`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
11`GoogleProducer`	Google says user-triggered fetchers generally ignore robots.txt rules. Google documents no robots.txt token for user-triggered fetchers; the token field holds the product name from the user agent string.
12`meta-externalfetcher`	Meta says it may bypass robots.txt because it performs fetches requested by the user.
13`ChatGPT-User`	OpenAI says robots.txt rules may not apply because these actions are user-initiated, and names only OAI-SearchBot and GPTBot as robots.txt tags. ChatGPT-User is not used to decide Search inclusion.
14`Perplexity-User`	Perplexity says this fetcher generally ignores robots.txt because a user requested the fetch, although the same page lists Perplexity-User among the robots.txt tags webmasters can use.

AI opt-outs

These tell Google, Apple and Yandex whether some of their AI may use pages their other bots collect: Gemini for Google, AI training for Apple and AI answers in search for Yandex. Blocking them does not stop these companies' search bots from reading your pages.

Crawler	What it does	Follows robots.txt	Verify by
`Applebot-Extended`Apple	Secondary robots.txt user agent that lets publishers opt out of their content being used to train Apple's foundation models; it does not crawl pages. Details for Applebot-Extended User agent Not documented robots.txt It is itself a robots.txt control token. Pages that disallow it can still appear in Apple search results. For AI answers to broad questions in Siri and Search, Apple points to the nosnippet meta tag instead. Verification No separate crawler to verify; Apple says it only determines how data crawled by Applebot is used. Sources About Applebot Checked October 15, 2026	Yes	Not documented
`Google-Extended`Google	Robots.txt token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI; it does not affect Google Search. Details for Google-Extended User agent None of its own. Google crawls with its existing user agent strings; the token only controls use. robots.txt It is itself a robots.txt control token. Google says it has no separate user agent; crawling uses existing Google user agents. Verification No separate crawler to verify; requests come from existing Google crawlers. Sources Google's common crawlers, read October 14, 2026 Checked October 15, 2026	Yes	Not documented
`YandexAdditionalBot`YandexThen `YandexAdditional`Ignores `*`	Handles robots.txt rules that keep page content out of Search with Yandex AI answers; Yandex says it makes no indexing requests, though it has user agent strings of its own. Details for YandexAdditionalBot User agent `Mozilla/5.0 (compatible; YandexAdditionalBot/1.0; +http://yandex.com/bots)` `Mozilla/5.0 (compatible; YandexAdditional/1.0; +http://yandex.com/bots)` robots.txt Yandex says to disallow YandexAdditional or YandexAdditionalBot to keep content out of Yandex AI answers; the tester tries both. Its table says it does not take the general (*) rules into account. Verification Reverse DNS hostname must end in yandex.ru, yandex.net or yandex.com, confirmed with a forward lookup. Yandex does not disclose its IP list. Sources Search with Yandex AI, read October 14, 2026 How to check that a robot belongs to Yandex, read October 14, 2026 Checked October 15, 2026	Yes	Reverse DNS: `yandex.ru`, `yandex.net`, `yandex.com`

Other bots

These do jobs outside the groups above, such as checking ads, making link previews, building open web datasets or running testing tools.

Crawler	What it does	Follows robots.txt	Verify by
`iTMS`AppleNo documented robots.txt token	Crawls only URLs associated with registered content on Apple Podcasts. Details for iTMS User agent `iTMS` Verification Apple says iTMS traffic may come from applebot.apple.com hosts, but documents no way to verify it. Sources About Applebot, read October 14, 2026 Checked October 15, 2026	NoNote 1	Not documented
`CCBot`Common Crawl	Crawler for Common Crawl, a non-profit that maintains an open repository of web crawl data anyone can access and analyze; the CCBot page does not describe it as an AI training crawler. Details for CCBot User agent `CCBot/2.0 (https://commoncrawl.org/faq/)` robots.txt CCBot checks robots.txt before fetching, obeys Crawl-delay, and can be blocked with a User-agent: CCBot group. Verification Reverse DNS on CCBot's dedicated ranges resolves to *.crawl.commoncrawl.org (not yet for IPv6); confirm with a forward lookup, or match the IP against Common Crawl's JSON list. Sources CCBot, read October 14, 2026 FAQ, read October 14, 2026 Checked October 15, 2026	Yes	IP list Reverse DNS: `crawl.commoncrawl.org`
`AdsBot-Google`GoogleIgnores `*`	Checks web page ad quality for Google Ads. Details for AdsBot-Google User agent `AdsBot-Google (+http://www.google.com/adsbot.html)` Verification Special-case crawlers: reverse DNS matches rate-limited-proxy-*---*.google.com; confirm with a forward DNS lookup, or match the IP against special-crawlers.json. Sources List of Google's special-case crawlers Checked October 15, 2026	PartlyNote 2	IP list Reverse DNS: `google.com`
`AdsBot-Google-Mobile`GoogleIgnores `*`	Mobile variant of AdsBot that checks web page ad quality for Google Ads. Details for AdsBot-Google-Mobile User agent `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)` Verification Special-case crawlers: reverse DNS matches rate-limited-proxy-*---*.google.com; confirm with a forward DNS lookup, or match the IP against special-crawlers.json. Sources List of Google's special-case crawlers Checked October 15, 2026	PartlyNote 3	IP list Reverse DNS: `google.com`
`APIs-Google`GoogleIgnores `*`	Used by Google APIs to deliver push notification messages. Details for APIs-Google User agent `APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)` Verification Special-case crawlers resolve to rate-limited-proxy-*---*.google.com; Google's APIs-Google page also names googlebot.com. Confirm with a forward DNS lookup, or match the IP against special-crawlers.json. Sources List of Google's special-case crawlers APIs-Google user agent Checked October 15, 2026	PartlyNote 4	IP list Reverse DNS: `google.com`, `googlebot.com`
`Google-CloudVertexBot`GoogleThen `Googlebot`	Crawls sites at the site owner's request to build Vertex AI Agents; Google says it has no effect on Google Search. Details for Google-CloudVertexBot User agent Contains: `Google-CloudVertexBot` robots.txt Listed with the common crawlers. Google documents only a user agent substring (Google-CloudVertexBot), and lists both Google-CloudVertexBot and Googlebot as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers, read October 14, 2026 Checked October 15, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Google-InspectionTool`GoogleThen `Googlebot`	Fetcher used by Search testing tools such as the Rich Result Test and URL inspection in Search Console; Google says it has no effect on Google Search. Details for Google-InspectionTool User agent Desktop: `Mozilla/5.0 (compatible; Google-InspectionTool/1.0)` Mobile: `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)` robots.txt Listed with the common crawlers, which obey robots.txt for automatic crawls. Google lists both Google-InspectionTool and Googlebot as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers, read October 14, 2026 Checked October 15, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Google-Safety`GoogleNo documented robots.txt token	Handles abuse-specific crawling, such as malware discovery for links posted publicly on Google properties. Details for Google-Safety User agent `Google-Safety` Verification Special-case crawlers: reverse DNS matches rate-limited-proxy-*---*.google.com; confirm with a forward DNS lookup, or match the IP against special-crawlers.json. Sources Google Special-Case Crawlers, read October 14, 2026 Checked October 15, 2026	NoNote 5	IP list Reverse DNS: `google.com`
`GoogleOther`Google	Generic Google crawler that product teams may use to fetch public content, for example one-off crawls for internal research and development. Details for GoogleOther User agent `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)` `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36` robots.txt Common crawler; obeys robots.txt for automatic crawls. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers, read October 14, 2026 Checked October 15, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`GoogleOther-Image`GoogleThen `GoogleOther`	Version of GoogleOther for fetching publicly accessible image URLs; not tied to a specific product. Details for GoogleOther-Image User agent `GoogleOther-Image/1.0` robots.txt Common crawler; Google lists both GoogleOther-Image and GoogleOther as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers Checked October 14, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`GoogleOther-Video`GoogleThen `GoogleOther`	Version of GoogleOther for fetching publicly accessible video URLs; not tied to a specific product. Details for GoogleOther-Video User agent `GoogleOther-Video/1.0` robots.txt Common crawler; Google lists both GoogleOther-Video and GoogleOther as its tokens. Verification Common crawlers: reverse DNS matches crawl-*---.googlebot.com or geo-crawl----*.geo.googlebot.com; confirm with a forward DNS lookup, or match the IP against common-crawlers.json. Sources Google's common crawlers Checked October 14, 2026	Yes	IP list Reverse DNS: `googlebot.com`
`Mediapartners-Google`GoogleIgnores `*`	Visits participating sites to serve relevant ads in Google ad products such as AdSense and Ad Manager. Details for Mediapartners-Google User agent Desktop: `Mediapartners-Google` Mobile: `(Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)` Verification Special-case crawlers: reverse DNS matches rate-limited-proxy-*---*.google.com; confirm with a forward DNS lookup, or match the IP against special-crawlers.json. Sources List of Google's special-case crawlers Checked October 15, 2026	PartlyNote 6	IP list Reverse DNS: `google.com`
`facebookexternalhit`Meta	Fetches apps or websites shared on Meta's apps (Facebook, Instagram, Messenger) to show the title, description and thumbnail. Details for facebookexternalhit User agent `facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)` Verification Meta's page says to allow-list the user agent strings or IP addresses but does not publish an IP list or reverse DNS method. Sources Meta Web Crawlers Checked October 14, 2026	PartlyNote 7	Not documented
`meta-externalads`Meta	Crawls the web for uses such as improving advertising and other business-related products and services. Details for meta-externalads User agent `meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers)` robots.txt Meta says to block its crawlers with a robots.txt disallow. Verification Meta publishes no IP list or reverse DNS method for this crawler. Sources Meta Web Crawlers, read October 14, 2026 Checked October 15, 2026	Yes	Not documented
`AdIdxBot`MicrosoftNo documented robots.txt tokenFollows `*`	Bing Ads crawler that crawls ads and follows the linked websites for quality control. Details for AdIdxBot User agent `Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)` robots.txt Bing says robots.txt controls its crawlers and that a User-agent: * section applies to all bots. It documents no AdIdxBot token, so the tester applies the * group. Verification Reverse DNS lookup should end in search.msn.com, then a forward lookup must return the same IP; or match the IP against bingbot.json (Bing says to refresh it daily). Bing also offers a Verify Bingbot tool. Sources Overview of Bing crawlers (user agents), read October 14, 2026 How to create a robots.txt file Checked October 15, 2026	Yes	IP list Reverse DNS: `search.msn.com`
`BingPreview`MicrosoftNo documented robots.txt tokenFollows `*`	Generates page snapshots for Bing; the user agent strings Bing lists for it contain bingbot/2.0. Details for BingPreview User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36` robots.txt Bing says robots.txt controls its crawlers and that a User-agent: * section applies to all bots. It documents no BingPreview token, so the tester applies the * group. Verification Reverse DNS lookup should end in search.msn.com, then a forward lookup must return the same IP; or match the IP against bingbot.json (Bing says to refresh it daily). Bing also offers a Verify Bingbot tool. Sources Overview of Bing crawlers (user agents), read October 14, 2026 How to create a robots.txt file Checked October 15, 2026	Yes	IP list Reverse DNS: `search.msn.com`
`BingVideoPreview`MicrosoftNo documented robots.txt tokenFollows `*`	Provides previews of videos in Bing. Details for BingVideoPreview User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; BingVideoPreview/1.0; +https://aka.ms/microsoftbots) Chrome/W.X.Y.Z Safari/537.36` robots.txt Bing says robots.txt controls its crawlers and that a User-agent: * section applies to all bots. It documents no BingVideoPreview token, so the tester applies the * group. Verification Reverse DNS lookup should end in search.msn.com, then a forward lookup must return the same IP; or match the IP against bingbot.json (Bing says to refresh it daily). Bing also offers a Verify Bingbot tool. Sources Overview of Bing crawlers (user agents), read October 14, 2026 How to create a robots.txt file Checked October 15, 2026	Yes	IP list Reverse DNS: `search.msn.com`
`MicrosoftPreview`MicrosoftNo documented robots.txt tokenFollows `*`	Generates page snapshots for Microsoft products. Details for MicrosoftPreview User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MicrosoftPreview/2.0; +https://aka.ms/MicrosoftPreview) Chrome/W.X.Y.Z Safari/537.36` robots.txt Bing says robots.txt controls its crawlers and that a User-agent: * section applies to all bots. It documents no MicrosoftPreview token, so the tester applies the * group. Verification Reverse DNS lookup should end in search.msn.com, then a forward lookup must return the same IP; or match the IP against bingbot.json (Bing says to refresh it daily). Bing also offers a Verify Bingbot tool. Sources Overview of Bing crawlers (user agents), read October 14, 2026 How to create a robots.txt file Checked October 15, 2026	Yes	IP list Reverse DNS: `search.msn.com`
`OAI-AdsBot`OpenAINo documented robots.txt token	Checks the safety of landing pages submitted as ads on ChatGPT and may use page content for ad relevance; OpenAI says it only visits pages submitted as ads. Details for OAI-AdsBot User agent `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot` Verification OpenAI publishes IP addresses for OAI-AdsBot; no reverse DNS method is documented. Sources Overview of OpenAI Crawlers, read October 14, 2026 Checked October 15, 2026	UnknownNote 8	IP list

Notes on robots.txt: other bots
Crawler	Note
1`iTMS`	Apple says the iTMS user agent does not follow robots.txt because it is not a general search crawler. Apple documents no robots.txt token for it.
2`AdsBot-Google`	Ignores the global user agent (*) with the ad publisher's permission; rules addressed to AdsBot-Google apply.
3`AdsBot-Google-Mobile`	Ignores the global user agent (*); rules addressed to AdsBot-Google-Mobile apply.
4`APIs-Google`	Ignores the global user agent (*); rules addressed to APIs-Google apply.
5`Google-Safety`	Google says this user agent ignores robots.txt rules.
6`Mediapartners-Google`	Ignores the global user agent (*); rules addressed to Mediapartners-Google apply.
7`facebookexternalhit`	Meta says it might bypass robots.txt when performing security or integrity checks.
8`OAI-AdsBot`	OpenAI documents no robots.txt token or rules for OAI-AdsBot. It says the bot visits only pages submitted as ads.