Short answer. Training crawlers and retrieval crawlers are different bots doing different jobs, and most blocking decisions treat them as one. Blocking a training crawler costs you nothing you can currently measure. Blocking a retrieval crawler makes citation impossible. The trap is that the common defaults do not distinguish them: since 1 July 2025, per Cloudflare's own documentation reported by Digital Applied, new domains on Cloudflare block GPTBot, ClaudeBot and PerplexityBot by default under a single toggle labelled for training.
Before any content is commissioned for AI visibility, one question decides whether the money is spendable at all: can the engines' crawlers fetch the pages? On a large share of sites they cannot, and nobody in the marketing team made that call. This piece sets out what each bot does, who is being blocked, what the crawl-to-referral economics actually look like, and a default policy that survives a review.
What each bot actually does
The user agent string is the whole decision. These are separate directives in robots.txt and can be set independently.
| User agent | Operator | Job | Blocking it costs you |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Builds the index ChatGPT search answers from | Any possibility of a ChatGPT citation |
| GPTBot | OpenAI | Collects training data | Presence in a future model's memory |
| ChatGPT-User | OpenAI | Fetches a page live during a user request | The page loading when a user asks about it |
| ClaudeBot | Anthropic | Collects training data | Presence in a future model's memory |
| Claude-SearchBot / Claude-User | Anthropic | Retrieval and live user fetches | Citation in Claude answers |
| PerplexityBot | Perplexity | Builds its retrieval index | Citation in Perplexity answers |
| Google-Extended | Controls Gemini training use only | Nothing in Search or AI Overviews | |
| Googlebot | Search index, and the basis of AI Overviews | Search and AI Overviews together |
Google-Extended is the one people get wrong in the expensive direction. It governs training use, not Search. Blocking it does not remove you from AI Overviews, because AI Overviews are built on the ordinary Googlebot crawl — and the same is true of AI Mode. There is no separate opt-out for AI Overviews that keeps you in Search.
The OpenAI split matters for the same reason in the other direction: ChatGPT cites from OAI-SearchBot's index, not from GPTBot's training crawl, so a site that blocks "OpenAI" as one thing loses the citable half along with the trainable half.
The default that blocks you without anyone deciding to
Cloudflare's default for new domains since 1 July 2025 blocks GPTBot, ClaudeBot and PerplexityBot, and the setting does not distinguish training crawlers from retrieval crawlers. PerplexityBot is purely a retrieval crawler; it is caught by a control labelled for training.
This is the most common cause of a site being invisible to AI answers for reasons nobody chose. A site can be perfectly written, perfectly structured, fully schema-marked and completely uncitable because of an infrastructure default set on the day the domain was onboarded.
Four checks, in the order they fail:
- Fetch
/robots.txtand read it. Look forDisallowunder each AI user agent by name, not just under*. - Check the CDN or WAF layer separately. A permissive robots.txt means nothing if the edge returns 403 to the bot.
- Check server logs or CDN analytics for real hits from OAI-SearchBot, PerplexityBot and Googlebot. Permission is a claim; a 200 in the log is evidence.
- Check for JavaScript-dependent content. Retrieval crawlers are not guaranteed to execute it, and a page whose text arrives only after hydration may be an empty page to them.
Who is being blocked, and how much
Blocking is now common enough to be a market-shaping fact rather than an edge case. Technology Checker's July 2026 report parsed a snapshot of 4,223 robots.txt files taken on 27 July 2026:
| User agent | Domains disallowing it |
|---|---|
| GPTBot | 633 |
| CCBot | 567 |
| ClaudeBot | 563 |
| Google-Extended | 522 |
| Bytespider | 518 |
The same report split AI crawler traffic by declared purpose: 44.54% training, 39.99% mixed, 11.57% search and 2.66% user-initiated. Training and mixed-purpose crawling together account for 84.5% of AI crawler traffic; search and user-initiated fetches account for 14.2%.
That split is the crux of the whole argument. Most of what hits your server has no path back to you at all — but the minority that does is exactly the traffic a blanket block removes.
For volume, Cloudflare Radar data for May 2026, reported by Digital Applied, put AI crawlers at 20.3% of verified bot traffic, with AI-search bots adding a further 6.5%. GPTBot accounted for 11.48% of AI bot requests and ClaudeBot 9.73%, reversing April's order.
How much they take for what they return
The case for blocking is usually made on this number, so it is worth quoting honestly, including where the sources disagree. Pages crawled per referral sent back, July 2026:
| Operator | Technology Checker | Digital Applied, reading Cloudflare data |
|---|---|---|
| Anthropic / ClaudeBot | 1,917:1 | 11,122:1 |
| OpenAI / GPTBot | 251:1 | 1,276:1 |
| Perplexity | 289:1 | — |
| 4.7:1 | — |
The two sources disagree by an order of magnitude, and both are carried here rather than one being picked, because they measure different site panels. Neither is quoted as the ratio. What survives the disagreement is the shape: AI crawlers take far more than they return, and Google returns orders of magnitude more traffic per page crawled than any of them.
That is a genuine argument for blocking training crawlers, and publishers with real bandwidth costs and licensing leverage are right to make it. It is not an argument for blocking retrieval crawlers — which is the decision the common defaults actually make for you.
A defensible default policy
Most organisations selling something, rather than licensing content, land in roughly the same place.
- Allow every retrieval and user-action crawler. OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Googlebot. These are the only bots that can produce a citation or a visit. There is no upside to blocking them unless you are deliberately withholding content.
- Decide training crawlers on principle, not on traffic. GPTBot, ClaudeBot, CCBot, Google-Extended. Allowing them is a bet on being in a future model's memory, which is real but unmeasurable and slow. Blocking them costs nothing you can currently observe. Either answer is defensible; pick one and write down why.
- Block scrapers with no answer surface. Bytespider and similar bots that neither cite nor refer. Nothing is given up.
- Verify at the edge, not just in
robots.txt. Confirm the CDN, WAF and bot-management rules agree with the file. This is where the policy is usually contradicted. - Re-check quarterly. User agents get added, platform defaults change, and a security review can revert the whole thing in one commit. Crawler access is a state to be monitored, not a task to be completed.
Note the asymmetry in step 2. The cost of blocking a training crawler is unobservable; the cost of blocking a retrieval crawler is immediate and total for that engine. When a decision has one measurable side and one unmeasurable side, put the measurable side first.
Why llms.txt is not part of this
It comes up in every crawler conversation, so it is worth settling.
Google documentation updated in June 2026 states that llms.txt has no effect on Search rankings or AI Overviews, and John Mueller of Google Search Relations noted that no AI crawler has claimed it extracts information from the file. On adoption, an SE Ranking study of 300,000 domains found llms.txt on 10.13% of them, while an Ahrefs study of 137,000 sites found 97% of published llms.txt files received zero traffic in May 2026.
Publishing one is harmless. Counting it as AI visibility work is not, because it displaces the crawler-access check that actually decides whether you can be cited. A file no crawler reads cannot substitute for a Disallow line that every crawler obeys.
What none of this buys
Crawler access is a precondition, not a lever. Being fetchable makes citation possible; it does not make it likely, and nobody can guarantee a citation or a placement in any AI answer. The engines differ enormously in what they do with the pages they fetch — Perplexity's source list is far steadier than ChatGPT's, and Google's two surfaces disagree with each other. Blocking decisions are also not reversible retroactively: a page excluded from an index while a bot was blocked was not cited during that period, and re-crawling runs on the operator's schedule, not yours.
How Lifewood approaches this
Lifewood runs crawler access as the first gate on any AEO or GEO engagement, before content is scoped, because content work aimed at a page the engine cannot fetch has no possible return. The check is evidence-based rather than declarative: robots.txt read by user agent, edge and WAF rules read separately, and server logs inspected for actual 200 responses to OAI-SearchBot, PerplexityBot and Googlebot — permission in a file is a claim, a log line is proof.
The training-crawler question is treated as a client policy decision rather than a technical default, documented with its reasoning so a later security review does not silently reverse it. Rendering is checked in the same pass, since a retrieval crawler that executes no JavaScript reads an empty page regardless of what robots.txt permits. See AEO services, GEO services, AEO and GEO providers and what gets you cited by AI answer engines.
Sources and further reading
- Technology Checker, robots.txt AI crawler blocking report, 4,223 files parsed 27 July 2026.
- Digital Applied, AI crawler and bot traffic statistics 2026, reading Cloudflare Radar.
- Digital Applied, AI crawler access control: the 2026 decision matrix.
- Google Search Relations on llms.txt, June 2026, via Baseline Labs.
- SE Ranking and Ahrefs llms.txt adoption studies, reported by Digital Applied.