AI Crawler & robots.txt Checker
See which AI crawlers your robots.txt blocks — and which of those blocks quietly remove you from ChatGPT, Claude and Perplexity answers.
Training, retrieval, and the block that costs you
Most AI crawler advice treats “AI bots” as one category. They are not, and the distinction decides whether a robots.txt rule protects you or quietly removes you from the answers you were trying to appear in.
- Training crawlers —
GPTBot,ClaudeBot,CCBot,Google-Extended. They collect content to train models. Blocking them is an editorial decision about whether your writing becomes training data. It has no effect on whether an assistant can find and cite you. - Retrieval crawlers —
OAI-SearchBot,Claude-SearchBot,PerplexityBot. They build the index an assistant searches when someone asks a live question. Blocking one removes you from that assistant’s answers. - User-requested fetchers —
ChatGPT-User,Claude-User,Perplexity-User. These fetch a page because a person pasted your link or asked the assistant to read it. Blocking one means a reader who explicitly asked for your page is told it cannot be read.
The names are similar enough that a rule written for one routinely catches another. That is the single most common finding this tool reports.
Google-Extended is not Googlebot
Worth stating on its own, because it is the question people get wrong most expensively: Google-Extended does not crawl anything. It is a permission token that controls whether content Googlebot already fetched may be used for Gemini training and for grounding AI answers. Disallowing it does not affect Google Search indexing or your ranking. Applebot-Extended works the same way for Apple.
How robots.txt actually decides
Three rules explain nearly every surprising result, and this checker implements all three:
- The most specific group wins outright. If a file has a group naming
GPTBot, that group is used instead of theUser-agent: *group — not in addition to it. A blanketDisallow: /under*does not apply to any crawler you named elsewhere. - The longest matching path wins, regardless of line order, with
Allowtaking ties. SoDisallow: /plusAllow: /blog/blocks the site but permits the blog. - Absence of a rule is permission. An empty
Disallow:, a missing robots.txt, or a 404 all mean everything is allowed. That is why this tool reports “no robots.txt” as a neutral result rather than an error.
Consecutive User-agent: lines also share the rules beneath them, so two agents listed back to back are governed by one block.
A configuration that usually reflects what people mean
If the intent is “don’t train on my work, but do cite me,” the shape is to disallow the training agents by name and stay silent about the retrieval ones:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Silence is the point. Adding Allow: / for the retrieval crawlers achieves nothing on its own and can narrow a wildcard rule defined elsewhere in the file, so the safer form emits denials only.
What this cannot tell you
Reachability is a precondition, not a cause. A blocked crawler certainly cannot quote you; an allowed one is under no obligation to. If every crawler here is allowed and you still never appear in answers, the constraint is somewhere else — usually that pages have no self-contained passage worth lifting. The AI Visibility Checker measures that side.
Frequently asked questions
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot collects content to train OpenAI's models. OAI-SearchBot fetches and indexes pages so ChatGPT can cite them when answering live questions. They are separate crawlers with separate robots.txt tokens. Blocking GPTBot keeps your content out of training and has no effect on whether ChatGPT can find you; blocking OAI-SearchBot removes you from ChatGPT's live answers entirely.
- Does blocking Google-Extended remove me from Google Search?
- No. Google-Extended is a permission token rather than a crawler: it controls whether your content is used for Gemini training and for grounding AI answers. Googlebot handles Search indexing and is unaffected. You can block Google-Extended and still rank normally in Google Search results.
- How do I block AI crawlers without blocking AI search?
- Disallow the training agents and leave the retrieval agents alone. In practice that means rules for GPTBot, ClaudeBot, Google-Extended, CCBot and the other training collectors, while allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot. The reverse — a blanket rule catching every agent with 'bot' or 'AI' in the name — is the configuration this tool most often finds.
- Why does a User-agent block sometimes override my wildcard rule?
- Because that is how robots.txt works. A group naming a specific crawler is used instead of the User-agent: * group, not in addition to it. A file with 'User-agent: *' 'Disallow: /' followed by 'User-agent: GPTBot' 'Allow: /' does not block GPTBot — the specific group wins outright. This checker follows the same precedence rules real crawlers do, and shows you which line decided each verdict.
- I have no robots.txt at all. Is that bad?
- Not for this purpose. Absence of a rule is permission, so a site with no robots.txt allows every crawler. A missing file is only a problem if you actually wanted to exclude something.
- Does this tell me whether ChatGPT will cite my site?
- No, and no tool can. This checks reachability, which is a precondition rather than a cause: a blocked crawler cannot cite you, but an allowed one is under no obligation to. Being reachable is the floor, not the ceiling.
Reachability is the floor. What decides whether a reachable page actually gets quoted is a separate question — this guide covers what the published research found.