Thirteen crawlers decide whether models can read you.
The registry, and what each block costs.
A crawler list without consequences is trivia. The column that decides anything is the last one: what you give up by writing a Disallow. Most robots.txt files that block AI agents were copied from a blog post, and the person who pasted them has no idea which of these thirteen lines they just switched off.
| Crawler | Operator | What it is for | What blocking it costs you |
|---|---|---|---|
GPTBot | OpenAI | Model training | Excluded from OpenAI training data |
OAI-SearchBot | OpenAI | ChatGPT Search index | No ChatGPT Search citations at all |
ChatGPT-User | OpenAI | Live fetch for a user | ChatGPT cannot open your page on request |
ClaudeBot | Anthropic | Model training | Excluded from Claude training data |
Claude-Web | Anthropic | User-triggered fetch | Claude cannot retrieve the page for a user |
anthropic-ai | Anthropic | Legacy training token | Excluded wherever the old name is honoured |
PerplexityBot | Perplexity | Cited answer index | Gone from Perplexity's citation list |
Google-Extended | Gemini and Vertex grounding | No Gemini grounding; Search unaffected | |
Bytespider | ByteDance | Training and product crawl | Excluded from ByteDance AI surfaces |
meta-externalagent | Meta | Meta AI training and answers | Excluded from Meta AI answers |
CCBot | Common Crawl | Open corpus others reuse | Removed from the dataset many models reuse |
cohere-ai | Cohere | Training and retrieval | Excluded from Cohere-derived assistants |
Applebot-Extended | Apple | Apple Intelligence training | No Apple Intelligence use; Siri unaffected |
The distinction that costs the most money.
“Should we block AI crawlers?” is not one question. It is at least two, and the two have opposite answers for most businesses. Training access is what a model learns from months or years before a user ever asks about you. Search and fetch access is what lets an assistant find you, quote you and link to you today.
GPTBot is not OAI-SearchBot
GPTBot crawls for model training. OAI-SearchBot crawls the index ChatGPT Search cites from. ChatGPT-User fetches a page live when a user’s prompt causes a browse. Blocking the first is a defensible position on training. Blocking the second removes you from ChatGPT’s citations entirely, which is almost never what the person writing the rule intended.
Google-Extended is not Googlebot
Google-Extended is a robots.txt opt-out token for Gemini grounding and Vertex AI generative features. It is not a crawler in its own right and it is not a ranking signal. Disallowing it removes you from Gemini grounding while leaving Google Search — AI Overviews included, since those are served from the ordinary Googlebot index — completely untouched.
Applebot-Extended is not Applebot
The same split again, from a different vendor: Applebot serves Siri and Spotlight search, while Applebot-Extended is the opt-out for training Apple Intelligence. Blocking the extended token leaves Apple’s search surfaces working. Blocking the base one does not.
One Disallow can close everything
A single over-broad rule can remove a site from every AI surface at once — and the symptom is invisible in ordinary rank tracking, because rankings do not move. The loss shows up only as citations that never arrive, which is the hardest kind of loss to notice and the easiest to leave running for a year.
Your robots.txt, read the way Google reads it.
Precedence is not first-rule-wins and it is not last-rule-wins. Google evaluates every rule whose path pattern matches the request and applies the longest one; when an Allow and a Disallow match at the same length, the Allow wins. That is why a file can look restrictive and be permissive, or the reverse, and why reading it by eye is unreliable.
| Request path | Rules that match | Longest match | Result |
|---|---|---|---|
/blog/geo-guide | Disallow: / + Allow: /blog/ | /blog/ — 6 characters | Allowed |
/app/settings | Disallow: / + Allow: /blog/ | / — 1 character | Blocked |
/blog/draft-7 | Allow: /blog/ + Disallow: /blog/draft | /blog/draft — 11 characters | Blocked |
/blog/ | Allow: /blog/ + Disallow: /blog/draft | /blog/ — 6 characters | Allowed |
/x | Allow: /x + Disallow: /x | equal length — Allow wins | Allowed |
The sentence Growzo will not soften
We never claim a crawler obeys anything. robots.txt is voluntary; this reports what the file INSTRUCTS. No tool can measure compliance from outside, and one that presents an instruction as an observed fact has quietly swapped a guess for a measurement.
The AI Crawler Blocking Index.
Growzo publishes its own longitudinal research on what the web actually instructs AI crawlers to do, built from repeated crawler-policy snapshots and shipped with a downloadable data.csv so the numbers can be checked rather than believed. It lives at /index/ai-crawler-blocking inside the Growzo app. What makes it worth citing is not the sample size — it is the four invariants encoded into how the numbers may be reported.
Unreachable is not blocked
A site that timed out, moved or returned a 500 has told you nothing about its crawler policy. Counting it as blocking inflates every blocking rate ever published, and it is the most common flaw in this genre of research.
A zero denominator yields null
Zero observations divided into anything is not zero percent — it is not a number. The index returns null and renders it as unmeasured, rather than drawing a reassuring line along the bottom of a chart.
Nothing is interpolated
Gaps between snapshots stay gaps. A smooth curve drawn through missing observations is a picture of the smoothing function, not of the web.
One run is never a trend
A single snapshot is a measurement of one moment. The index refuses to describe direction until it has enough runs to have seen one, which is the discipline the word “trend” is supposed to carry.
llms.txt, and the part nobody else ships.
Plenty of tools will generate an llms.txt for you. Growzo parses, validates and generates them — and then also serves them: live per-customer llms.txt and llms-full.txt endpoints that carry your whole published corpus as tag-free, attributable prose. Generating a file is a formatting exercise. Delivering the text is what actually puts readable content in front of a model that has already decided to look.
One honest caveat, because it belongs here: llms.txt is a proposal, not a ranking mechanism, and Google has said it does not use it. It is worth shipping because it is cheap and because some clients do read it — not because it is a lever. The GEO tier covers what is actually measurable.
Four sub-scores, over ten pages. Exactly ten.
The audit returns four sub-scores over a sample capped at ten pages— for a project audit and for the public audit alike. That ceiling is stated here rather than buried, because the gap between “we audited your site” and “we audited ten of its pages” is the whole difference between a finding and an impression. Pages are fetched through an SSRF-guarded crawler that reads robots.txt and sitemap.xml, and findings come back with a severity and a 0-100 score.
A sample described as a sample
Ten pages will find a robots.txt mistake, a missing canonical pattern or a template with no structured data, because those are class defects that repeat. Ten pages will not find the one broken URL in a thousand, and Growzo does not pretend otherwise.
SSRF-guarded fetching
The crawler refuses to be pointed at internal addresses. An auditing tool that will fetch any URL you hand it is a request-forgery gadget wearing a friendly name, and the guard is the reason this one is safe to run against a domain you do not own.
Crawler access is the cheapest thing on this page to get right and the most expensive to get wrong, which is why it comes first in the audit. If you want the spend side of the same story, the hard money cap explains what every probe and fetch costs before it happens.
Crawler questions, answered plainly.
Does blocking GPTBot stop ChatGPT from citing my site?
What does blocking Google-Extended do to my Google rankings?
Does Growzo guarantee that a crawler obeys robots.txt?
What is the AI Crawler Blocking Index?
What does Growzo do with llms.txt?
How many pages does the AI-readiness audit check?
Find out what your file is really telling them.
Growzo runs on its own application and its own sign-in — an Autocloz account is not required. Point it at your robots.txt and read the thirteen answers, agent by agent.