Technical guide
robots.txt for AI bots: what to block, what to allow, and a ready-to-copy example
AI bots come in three kinds: training crawlers (GPTBot, ClaudeBot), search crawlers that let AI cite you (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and agents that fetch a page because a user asked. For GEO, always allow the search ones; blocking training is optional. GeoRush checks your robots.txt and generates one for you.
Updated on
Why the difference between bots matters
Many sites blocked “AI bots” wholesale to keep their content out of model training. The problem is that in doing so they often also blocked the crawlers that feed the search-based answers in ChatGPT, Claude and Perplexity, and vanished from them.
Today the major companies split their agents by purpose, so you can decide precisely. The practical rule for GEO: allow search crawlers and user agents; decide on training separately.
The three kinds of AI bots
Plus a fourth special case: control tokens, which do not crawl at all.
Training crawlers
Collect content that may be used to train models. Blocking them does not remove you from search-based answers. Examples: GPTBot, ClaudeBot, CCBot, MistralAI-Training.
Search crawlers
Index pages so the assistant can find and cite them. Block them and you cannot be a source. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot.
User-triggered agents
Visit a page because someone asked right then. Some may not follow robots.txt. Examples: ChatGPT-User, Claude-User, Perplexity-User.
Control tokens
They do not crawl; they only state how already-crawled content may be used. Examples: Google-Extended, Applebot-Extended.
AI user agents and what each one does
Taken from each company’s official documentation, as of November 2026. Names and purposes change: check the sources at the end before making changes.
| Agent (user-agent) | Company | Type | What it does |
|---|---|---|---|
GPTBot | OpenAI | Training | Crawls content that may be used to train OpenAI’s models. |
OAI-SearchBot | OpenAI | Search | Indexes sites to show them in ChatGPT search. Block it and you will not appear in those answers. |
ChatGPT-User | OpenAI | User | Visits pages for user actions in ChatGPT and GPTs. OpenAI says robots.txt may not apply. |
ClaudeBot | Anthropic | Training | Collects content that may contribute to training Anthropic’s models. Supports Crawl-delay. |
Claude-SearchBot | Anthropic | Search | Navigates the web to improve the quality of Claude’s search results. |
Claude-User | Anthropic | User | Accesses websites when a Claude user asks a question. Blocking it may reduce your visibility in user-directed search. |
PerplexityBot | Perplexity | Search | Surfaces and links websites in Perplexity results. Perplexity says it is not used to train foundation models. |
Perplexity-User | Perplexity | User | Visits a page to answer a user’s question. Perplexity says it generally ignores robots.txt. |
Googlebot | Search | Google Search’s crawler. AI Overviews and AI Mode depend on it too. | |
Google-Extended | Control token | Does not crawl. Controls whether Google uses your content to train Gemini and for grounding in Gemini Apps and Vertex AI. Does not affect Google Search. | |
Bingbot | Microsoft | Search | Bing’s crawler. OpenAI names Bing among ChatGPT’s search providers. |
Applebot | Apple | Search | Powers search features in Spotlight, Siri and Safari; Apple says this data may also be used to train its models. |
Applebot-Extended | Apple | Control token | Does not crawl. Lets you exclude your content from Apple model training while staying in its search. |
meta-externalagent | Meta | Training | Crawls to train AI models or improve products by indexing content. |
meta-webindexer | Meta | Search | Improves Meta AI search results; allowing it helps you get cited and linked. |
meta-externalfetcher | Meta | User | Fetches specific links at a user’s request; Meta says it may bypass robots.txt. |
MistralAI-Training | Mistral AI | Training | Builds datasets to train Mistral’s models. |
MistralAI-Index | Mistral AI | Search | Indexes content for search in Mistral’s assistants. |
MistralAI-User | Mistral AI | User | Fetches pages when a user asks a question and links the source in the answer. |
CCBot | Common Crawl | Training | Builds an open web archive that many organizations use to train models. |
A ready-to-copy robots.txt example
This version allows every search crawler and user agent and blocks the training ones. Copy it, change the sitemap domain and adapt the private paths. If you also want to allow training, delete the second group. We left Google-Extended out on purpose, for the reason above.
Important: under the standard (RFC 9309), a bot that finds a group with its own name follows only that group and ignores User-agent: *. That is why private paths are repeated in the first group.
# Search crawlers and user agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
Disallow: /admin/
# Model training: blocked (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /
# All other bots
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.your-domain.com/sitemap.xml
How to roll it out without breaking anything
- 1
Review your current robots.txt
Open
your-domain.com/robots.txtand look for rules that affect search bots, including a blanketDisallow: /underUser-agent: *. - 2
Check your CDN and firewall
Some CDNs and security services offer one-click AI bot blocking, and in some cases enable it by default. That blocking does not show up in robots.txt; check it in their dashboard.
- 3
Publish the file at the root
It must sit at the root of each host (subdomains included), be served as plain text and return a 200 status.
- 4
Wait and verify
The standard lets crawlers cache robots.txt for up to 24 hours; OpenAI and Meta mention about 24 hours. Then check your server logs to see which bots come in.
How GeoRush helps
The AI technical section of the SEO and technical audit checks which AI crawlers your robots.txt allows or blocks, whether you have llms.txt and structured data, and includes ready-to-paste generators for robots.txt, llms.txt and JSON-LD. Web analytics records AI crawler hits and the traffic arriving from assistants, so you can see whether your changes take effect.
The free check includes this robots.txt review, with no account needed.
Is your robots.txt hiding you from AI?
The free check reviews which AI bots you let in and tells you what to change.
Sources
Official documentation and primary sources checked on November 16, 2026. These policies change often: check the source before making important decisions.
- RFC 9309: Robots Exclusion Protocol (IETF)
- OpenAI: overview of OpenAI crawlers
- Anthropic: does Anthropic crawl data from the web and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: common crawlers and Google-Extended
- Google Search Central: AI features and your website
- Apple: about Applebot
- Meta: web crawlers
- Mistral AI: robots and crawlers
- Common Crawl: CCBot
Frequently asked questions
Keep reading
What is llms.txt
What llms.txt is, how to write one with a ready example, and what to expect from it, without the hype.
Appear in ChatGPT
Step by step to get ChatGPT to mention your brand and cite your site, and how to check it.
GEO glossary
34 GEO terms clearly defined: concepts, metrics, crawlers and technical data.
SEO and technical audit
Crawl your site, find SEO errors and check whether AI bots can read it, with code to fix it.