robots.txt examples for AI and search bots

Put the file you choose at your site's address followed by /robots.txt, such as example.com/robots.txt. Then type your website into the robots.txt checker to see the result.

Allow every bot

Every bot that follows robots.txt may read every page. A robots.txt address that returns 404 has the same effect: RFC 9309 lets crawlers read everything then.

User-agent: *
Allow: /

What it does for example.com's home page

Google: Allowed No robots.txt rule blocks this page for Googlebot.
Bing and Copilot: Allowed No robots.txt rule blocks this page for Bingbot.
ChatGPT: Allowed No robots.txt rule blocks this page for OAI-SearchBot.
Claude: Allowed No robots.txt rule blocks this page for Claude-SearchBot.
Perplexity: Allowed No robots.txt rule blocks this page for PerplexityBot.
Gemini: Allowed No robots.txt rule stops Google from using this page for Gemini (Google-Extended).

We have not checked whether the server lets these crawlers through.

AI training: none of 5 blocked.

Across all 65 bots in the crawler list: 51 allowed, 0 blocked and 14 not decided by robots.txt. Open this file in the robots.txt tester to see each one.

Block AI training bots, keep search and AI search

This keeps out the bots in the crawler list's AI training section. Their operators say the pages they collect may be used to train AI models. The file still allows Google, Bing and AI search bots.

User-agent: Amazonbot
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Allow: /

What it does for example.com's home page

Google: Allowed No robots.txt rule blocks this page for Googlebot.
Bing and Copilot: Allowed No robots.txt rule blocks this page for Bingbot.
ChatGPT: Allowed No robots.txt rule blocks this page for OAI-SearchBot.
Claude: Allowed No robots.txt rule blocks this page for Claude-SearchBot.
Perplexity: Allowed No robots.txt rule blocks this page for PerplexityBot.
Gemini: Allowed No robots.txt rule stops Google from using this page for Gemini (Google-Extended).

We have not checked whether the server lets these crawlers through.

AI training: 5 of 5 blocked.

Across all 65 bots in the crawler list: 46 allowed, 5 blocked and 14 not decided by robots.txt. Open this file in the robots.txt tester to see each one.

Common Crawl's CCBot is not in that section, since its own page does not describe it as an AI training crawler. To keep it out too, add User-agent: CCBotto the first group. This file does not stop Google or Apple from using pages their search bots collect for AI. The next example opts out of some of those uses, such as training Gemini and Apple's foundation models.

Opt out of some AI uses by Google, Apple and Yandex, keep their search

Google-Extended and Applebot-Extended do not crawl pages, and Yandex says YandexAdditionalBot makes no indexing requests. They tell Google, Apple and Yandex how they may use pages their search bots collect:

Google-Extended: training future Gemini models, and grounding in Gemini Apps and Vertex AI
Applebot-Extended: training Apple's foundation models
YandexAdditionalBot: Yandex's AI answers in search

User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: YandexAdditionalBot
Disallow: /

User-agent: *
Allow: /

What it does for example.com's home page

Google: Allowed No robots.txt rule blocks this page for Googlebot.
Bing and Copilot: Allowed No robots.txt rule blocks this page for Bingbot.
ChatGPT: Allowed No robots.txt rule blocks this page for OAI-SearchBot.
Claude: Allowed No robots.txt rule blocks this page for Claude-SearchBot.
Perplexity: Allowed No robots.txt rule blocks this page for PerplexityBot.
Gemini: Blocked Line 4 asks Google not to use this page for Gemini (Google-Extended): Disallow: /

We have not checked whether the server lets these crawlers through.

AI training: none of 5 blocked.

Across all 65 bots in the crawler list: 48 allowed, 3 blocked and 14 not decided by robots.txt. Open this file in the robots.txt tester to see each one.

Google says Google-Extended does not affect inclusion in Google Search. Apple says pages that disallow Applebot-Extended can still appear in its search results.

Allow search engines only

Only the search engine bots you name, and bots that follow their rules, may read your pages. Every other bot that follows robots.txt is blocked, including AI search and AI training bots. Five bots skip the *group, so they stay allowed (see Block every bot).

User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckDuckBot
Allow: /

User-agent: *
Disallow: /

What it does for example.com's home page

Google: Allowed No robots.txt rule blocks this page for Googlebot.
Bing and Copilot: Allowed No robots.txt rule blocks this page for Bingbot.
ChatGPT: Blocked Line 8 blocks this page for OAI-SearchBot: Disallow: /
Claude: Blocked Line 8 blocks this page for Claude-SearchBot: Disallow: /
Perplexity: Blocked Line 8 blocks this page for PerplexityBot: Disallow: /
Gemini: Blocked Line 8 asks Google not to use this page for Gemini (Google-Extended): Disallow: /

We have not checked whether the server lets these crawlers through.

AI training: 5 of 5 blocked.

Across all 65 bots in the crawler list: 14 allowed, 35 blocked, 2 depend on the operator's reading and 14 not decided by robots.txt. Open this file in the robots.txt tester to see each one.

Google says AI features such as AI Overviews are built into Search, and robots.txt rules for Googlebot control how a site is crawled for Search.Bing says Copilot's search relies on the same crawling as its search.

Googlebot's group also lets in the bots that follow its rules: Brave Search crawler, Google-CloudVertexBot, Google-InspectionTool, Googlebot-Image, Googlebot-News and Googlebot-Video. Other search bots you do not name are blocked, such as YandexBot, Baiduspider and Storebot-Google, which Google uses for Shopping. Add the ones you need to the first group.

Two results depend on the operator's reading, as the method explains.Apple's own example has Applebot obey the *group too.Amazon says Amzn-SearchBot, with no group of its own, follows other search bots' rules if the file allows them.

Keep bots out of one folder

Bots that follow robots.txt stay out of every address that starts with /private/, such as /private/report.html. The rest of the site stays open to them. Five bots skip the * group and stay allowed (see Block every bot).

User-agent: *
Disallow: /private/

What it does for /private/report.html on example.com

Google: Blocked Line 2 blocks this page for Googlebot: Disallow: /private/
Bing and Copilot: Blocked Line 2 blocks this page for Bingbot: Disallow: /private/
ChatGPT: Blocked Line 2 blocks this page for OAI-SearchBot: Disallow: /private/
Claude: Blocked Line 2 blocks this page for Claude-SearchBot: Disallow: /private/
Perplexity: Blocked Line 2 blocks this page for PerplexityBot: Disallow: /private/
Gemini: Blocked Line 2 asks Google not to use this page for Gemini (Google-Extended): Disallow: /private/

We have not checked whether the server lets these crawlers through.

AI training: 5 of 5 blocked.

Across all 65 bots in the crawler list: 5 allowed, 46 blocked and 14 not decided by robots.txt. Open this file in the robots.txt tester to see each one.

Rules match from the start of the path, so this file does not cover /blog/private/.

robots.txt is not a lock. RFC 9309 says listing a path in it makes the path public, and suggests a password for anything private. Google says a blocked page can still appear in its results, without a description, when other pages link to it.

Block every bot

Bots that follow robots.txt stay out, apart from the five in the note below, as you might want on a test site.Google says blocked addresses can still appear in its results, without a description, when other sites link to them. A password keeps out people and bots alike.

User-agent: *
Disallow: /

What it does for example.com's home page

Google: Blocked Line 2 blocks this page for Googlebot: Disallow: /
Bing and Copilot: Blocked Line 2 blocks this page for Bingbot: Disallow: /
ChatGPT: Blocked Line 2 blocks this page for OAI-SearchBot: Disallow: /
Claude: Blocked Line 2 blocks this page for Claude-SearchBot: Disallow: /
Perplexity: Blocked Line 2 blocks this page for PerplexityBot: Disallow: /
Gemini: Blocked Line 2 asks Google not to use this page for Gemini (Google-Extended): Disallow: /

We have not checked whether the server lets these crawlers through.

AI training: 5 of 5 blocked.

Across all 65 bots in the crawler list: 5 allowed, 46 blocked and 14 not decided by robots.txt. Open this file in the robots.txt tester to see each one.

Five bots skip the * group, so this file leaves them allowed: AdsBot-Google, AdsBot-Google-Mobile, APIs-Google and Mediapartners-Google from Google, and YandexAdditionalBot. To block them too, give them a group of their own.