A site owner reads that AI companies are training on the open web, decides they would rather not participate, and adds a few lines to robots.txt. Six months later they notice they have disappeared from ChatGPT's answers while their competitors are being cited.
That outcome is not a policy decision going wrong. It is one crawler being mistaken for another. This guide is about telling them apart.
Three jobs, not one
"AI crawler" describes at least three different things, and the consequence of blocking one has nothing to do with the consequence of blocking another.
Training crawlers
These collect content that may be used to train models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose public dataset many models train on), Bytespider, meta-externalagent and Amazonbot are in this group.
Blocking them removes your content from training data and has no effect on whether an assistant can find you today. Whether you want to is a genuine editorial and commercial judgement, and there is an honest case on both sides. It is also the decision most people think they are making when they write an AI crawler rule.
Retrieval crawlers
These fetch and index pages so an assistant can search them when answering a live question. OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot (Perplexity) are in this group.
Blocking one removes you from that assistant's answers. Not from its training data — from the live results. This is almost never what someone intends, and it is the failure this guide exists to prevent.
User-requested fetchers
The category most guides omit. ChatGPT-User, Claude-User and Perplexity-User fetch a specific page because a human pasted a link or asked the assistant to go read it. There is no crawl and no index — it is one fetch, on demand, for someone who explicitly asked.
Blocking one means a reader who wanted your page is told it cannot be read. It is the closest equivalent to refusing a visitor who followed a link, and it is very rarely a considered choice.
The pairs that get confused
| Agent | Vendor | Blocking it means |
|---|---|---|
GPTBot | OpenAI | Excluded from model training. No effect on ChatGPT search. |
OAI-SearchBot | OpenAI | Removed from ChatGPT's live search results. |
ChatGPT-User | OpenAI | A user who asks ChatGPT to read your link is refused. |
ClaudeBot | Anthropic | Excluded from model training. |
Claude-SearchBot | Anthropic | Removed from Claude's live retrieval. |
PerplexityBot | Perplexity | Removed from Perplexity results. |
Google-Extended | Excluded from Gemini training and AI grounding. No effect on Google Search. | |
CCBot | Common Crawl | Excluded from a dataset many models train on. |
Read the OpenAI rows together. GPTBot and OAI-SearchBot differ by one word and produce opposite outcomes. The same is true of ClaudeBot and Claude-SearchBot.
Google-Extended is not a crawler
Worth its own section, because it is the most expensive misunderstanding here.
Google-Extended does not fetch anything. It is a permission token that controls whether content Googlebot has already crawled may be used for Gemini training and for grounding AI answers, per Google's crawler documentation. Googlebot is a separate agent and handles Search.
Disallowing Google-Extended does not affect your Google Search indexing, ranking or traffic. If you have avoided it out of fear of a ranking penalty, that fear is misplaced. Applebot-Extended works the same way for Apple: plain Applebot continues to power Siri and Spotlight.
The robots.txt rule that surprises people
Even with the right agents identified, robots.txt has one precedence rule that inverts results:
A group naming a specific crawler is used instead of the User-agent: * group, not in addition to it.
So this file does not block GPTBot:
User-agent: *
Disallow: /
User-agent: GPTBot
Allow: /GPTBot reads its own group, finds Allow: /, and crawls the site. The blanket rule never applies to it. This works in both directions, and it is why "I added a global disallow, surely that covers everything" is often wrong.
Two more rules worth knowing:
- The longest matching path wins, regardless of line order, with
Allowtaking ties.Disallow: /plusAllow: /blog/blocks the site but permits the blog. - Consecutive
User-agent:lines share the rules beneath them, so several agents listed back to back are governed by one block.
What to actually write
If the intent is "don't train on my work, but do cite me" — which is what most people mean — disallow the training agents by name and say nothing at all about the retrieval ones:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /The silence is deliberate. Adding Allow: / for OAI-SearchBot achieves nothing on its own — absence of a rule is already permission — and an unnecessary Allow can narrow a wildcard rule defined elsewhere in the file. Emit denials only.
If instead you want out of AI entirely, including the answers, block both groups and accept the trade knowingly. That is a legitimate position; it should just be the one you chose.
Checking what you actually have
Robots.txt files accumulate. Rules get added by plugins, security tools and previous developers, and the interaction between a wildcard group and a named one is not obvious by reading. Our AI Crawler & robots.txt Checker fetches a site's file, applies the same precedence rules real crawlers use, and reports each agent with the line that decided it — grouped by what the block actually costs.
Being reachable is the floor, not the ceiling
Getting this right does not get you cited. It removes one reason not to be. A page that every crawler can reach still needs a passage worth lifting — which is a different problem, covered in how answer engines choose which sources to quote.
Reachability is worth fixing first only because it is binary: no amount of good writing survives a Disallow.