Technical guide

robots.txt for AI bots: what to block, what to allow, and a ready-to-copy example

AI bots come in three kinds: training crawlers (GPTBot, ClaudeBot), search crawlers that let AI cite you (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and agents that fetch a page because a user asked. For GEO, always allow the search ones; blocking training is optional. GeoRush checks your robots.txt and generates one for you.

Updated on

Why the difference between bots matters

Many sites blocked “AI bots” wholesale to keep their content out of model training. The problem is that in doing so they often also blocked the crawlers that feed the search-based answers in ChatGPT, Claude and Perplexity, and vanished from them.

Today the major companies split their agents by purpose, so you can decide precisely. The practical rule for GEO: allow search crawlers and user agents; decide on training separately.

The three kinds of AI bots

Plus a fourth special case: control tokens, which do not crawl at all.

Training crawlers

Collect content that may be used to train models. Blocking them does not remove you from search-based answers. Examples: GPTBot, ClaudeBot, CCBot, MistralAI-Training.

Search crawlers

Index pages so the assistant can find and cite them. Block them and you cannot be a source. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot.

User-triggered agents

Visit a page because someone asked right then. Some may not follow robots.txt. Examples: ChatGPT-User, Claude-User, Perplexity-User.

Control tokens

They do not crawl; they only state how already-crawled content may be used. Examples: Google-Extended, Applebot-Extended.

AI user agents and what each one does

Taken from each company’s official documentation, as of November 2026. Names and purposes change: check the sources at the end before making changes.

Agent (user-agent)CompanyTypeWhat it does
GPTBotOpenAITrainingCrawls content that may be used to train OpenAI’s models.
OAI-SearchBotOpenAISearchIndexes sites to show them in ChatGPT search. Block it and you will not appear in those answers.
ChatGPT-UserOpenAIUserVisits pages for user actions in ChatGPT and GPTs. OpenAI says robots.txt may not apply.
ClaudeBotAnthropicTrainingCollects content that may contribute to training Anthropic’s models. Supports Crawl-delay.
Claude-SearchBotAnthropicSearchNavigates the web to improve the quality of Claude’s search results.
Claude-UserAnthropicUserAccesses websites when a Claude user asks a question. Blocking it may reduce your visibility in user-directed search.
PerplexityBotPerplexitySearchSurfaces and links websites in Perplexity results. Perplexity says it is not used to train foundation models.
Perplexity-UserPerplexityUserVisits a page to answer a user’s question. Perplexity says it generally ignores robots.txt.
GooglebotGoogleSearchGoogle Search’s crawler. AI Overviews and AI Mode depend on it too.
Google-ExtendedGoogleControl tokenDoes not crawl. Controls whether Google uses your content to train Gemini and for grounding in Gemini Apps and Vertex AI. Does not affect Google Search.
BingbotMicrosoftSearchBing’s crawler. OpenAI names Bing among ChatGPT’s search providers.
ApplebotAppleSearchPowers search features in Spotlight, Siri and Safari; Apple says this data may also be used to train its models.
Applebot-ExtendedAppleControl tokenDoes not crawl. Lets you exclude your content from Apple model training while staying in its search.
meta-externalagentMetaTrainingCrawls to train AI models or improve products by indexing content.
meta-webindexerMetaSearchImproves Meta AI search results; allowing it helps you get cited and linked.
meta-externalfetcherMetaUserFetches specific links at a user’s request; Meta says it may bypass robots.txt.
MistralAI-TrainingMistral AITrainingBuilds datasets to train Mistral’s models.
MistralAI-IndexMistral AISearchIndexes content for search in Mistral’s assistants.
MistralAI-UserMistral AIUserFetches pages when a user asks a question and links the source in the answer.
CCBotCommon CrawlTrainingBuilds an open web archive that many organizations use to train models.

A ready-to-copy robots.txt example

This version allows every search crawler and user agent and blocks the training ones. Copy it, change the sitemap domain and adapt the private paths. If you also want to allow training, delete the second group. We left Google-Extended out on purpose, for the reason above.

Important: under the standard (RFC 9309), a bot that finds a group with its own name follows only that group and ignores User-agent: *. That is why private paths are repeated in the first group.

robots.txt
# Search crawlers and user agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
Disallow: /admin/

# Model training: blocked (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# All other bots
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.your-domain.com/sitemap.xml

How to roll it out without breaking anything

  1. 1

    Review your current robots.txt

    Open your-domain.com/robots.txt and look for rules that affect search bots, including a blanket Disallow: / under User-agent: *.

  2. 2

    Check your CDN and firewall

    Some CDNs and security services offer one-click AI bot blocking, and in some cases enable it by default. That blocking does not show up in robots.txt; check it in their dashboard.

  3. 3

    Publish the file at the root

    It must sit at the root of each host (subdomains included), be served as plain text and return a 200 status.

  4. 4

    Wait and verify

    The standard lets crawlers cache robots.txt for up to 24 hours; OpenAI and Meta mention about 24 hours. Then check your server logs to see which bots come in.

How GeoRush helps

The AI technical section of the SEO and technical audit checks which AI crawlers your robots.txt allows or blocks, whether you have llms.txt and structured data, and includes ready-to-paste generators for robots.txt, llms.txt and JSON-LD. Web analytics records AI crawler hits and the traffic arriving from assistants, so you can see whether your changes take effect.

The free check includes this robots.txt review, with no account needed.

Is your robots.txt hiding you from AI?

The free check reviews which AI bots you let in and tells you what to change.

Sources

Official documentation and primary sources checked on November 16, 2026. These policies change often: check the source before making important decisions.

Frequently asked questions

To appear in AI answers, allow at least OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, plus user agents such as ChatGPT-User and Claude-User. Blocking training crawlers is a separate decision.

Not from ChatGPT search, which uses OAI-SearchBot. GPTBot only crawls content that may be used to train OpenAI’s models.

ClaudeBot collects content for training, Claude-SearchBot indexes to improve Claude’s search results, and Claude-User visits pages when a user asks a question. For GEO, you want to allow the last two.

Not entirely. It is a standard that well-behaved crawlers respect, but it is not access control. Some agents acting on a user’s request may also not follow it.

The standard allows caching for up to 24 hours, and OpenAI and Meta mention about 24 hours. Afterwards, check your server logs to confirm.

Keep reading