Short answer. By making sure Claude can reach your pages, then giving it something worth quoting. Access is the part most brands get wrong: Anthropic runs three separate crawlers with different jobs, and blocking the wrong one removes you from Claude's answers without affecting your Google rankings at all. Beyond access, the pattern in observed citations favours specific, sourced, well-structured content with visible dates over promotional copy.
Claude citation is not one optimisation problem. It is a discoverability problem and an access problem, and most brands work only on the first. This piece covers how Claude reaches a page, what each crawler does when you block it, what actually gets cited, what does not work, and how to measure it.
How does Claude actually reach a web page?
Through two routes: a search step that surfaces candidate pages, and a retrieval step that fetches them. Both have to work for a citation to happen.
Anthropic's own documentation states that it uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users' direction. Those are three distinct functions, and they fail independently.
On the search side, evidence reported in 2025 indicated that Claude's web search is powered by Brave Search, after Brave appeared on Anthropic's published subprocessor list and independent testing found matching citations. Third-party analysis has since reported high overlap between Claude's cited results and Brave's top organic results, which suggests visibility in Brave is a meaningful input to citation eligibility. Treat that as a well-supported inference rather than an official disclosure — Anthropic has not published its ranking signals.
Which crawler does what, and what happens if you block it?
Three bots, three different consequences. This is the highest-leverage thing on the page, and it is documented by Anthropic directly.
| Crawler | Job | What blocking it does |
|---|---|---|
| ClaudeBot | Collects web content that could potentially contribute to model training | Signals that your site's future materials should be excluded from training datasets. Affects background familiarity with your brand — not whether Claude can cite you in a live answer |
| Claude-User | Supports Claude users: accesses websites when someone asks a question | Prevents the system from retrieving your content in response to a user query, which may reduce visibility for user-directed web search |
| Claude-SearchBot | Navigates the web to improve search result quality | Prevents indexing of your content for search optimisation, which may reduce visibility and accuracy in user search results |
Anthropic also confirms its bots honour standard robots.txt directives, respect Crawl-delay where appropriate, and do not attempt to bypass anti-circumvention measures such as CAPTCHAs.
The failure mode to check for is a blanket AI-crawler block added at some point by a well-meaning engineer or a security vendor. It is entirely reasonable to exclude your content from model training while remaining fully available for live retrieval — disallow ClaudeBot, allow the other two. Blocking all three achieves something most marketing teams never intended, and nothing in your analytics will tell you it happened.
What kind of content gets cited?
Content that answers a question directly, states specifics, shows its sources and shows its date. Analyses of AI citation patterns converge on the same handful of properties.
- A direct answer near the top of each section. Answer engines lift passages, not pages. If the answer to the heading is three paragraphs down, the extracted passage may not contain it.
- Specific, attributable claims. A statistic with a named source and a date gives a model something concrete to quote. Vague superiority claims give it nothing.
- Descriptive, question-shaped headings. They let a retrieval system match a sub-question to a section, which matters because a single user query often generates several.
- Visible publish and update dates. Recency has been reported as a consistent factor in citation selection tests. Carry both a publish date and a genuine last-updated date — update the content first, then change the date.
- Neutral, factual register. A sentence a journalist could quote is more citable than a sentence a brochure would use.
- Depth across a topic, not a single page. Consistent coverage of one subject area builds the topical association that makes a site a reliable candidate rather than a lucky one.
That list describes the structure of the article you are reading. It is the same structure Lifewood applies through its AEO and GEO work: question-shaped headings, an answer block under each, sourced statistics, visible dates and schema markup, so a page is usable by an answer engine and by a human in the same pass.
What does not work?
Four things, two of which actively hurt.
- Keyword stuffing. Retrieval works on meaning, not term frequency. Repetition makes a passage less quotable, not more.
- Thin, generic articles. No original data, no first-hand experience, no clear position. If a hundred sites say the same thing, a model has no reason to name yours.
- Faking freshness. Changing a date without changing the content corrodes the trust signals you are trying to build.
- Blocking crawlers by accident. The only item here that can take you from visible to invisible overnight.
One further caution: no agency can guarantee AI citations, and any that does is selling something. Anthropic does not publish ranking signals, retrieval behaviour changes, and the honest framing is probabilistic — you are improving the likelihood that your page is the most useful available answer, not buying a placement.
How do you measure and improve citation visibility?
Build a prompt set, test it on a schedule, and track what gets cited instead of you.
- Write 20 to 50 real questions your customers would ask, in the phrasing they would use, including comparison and "best provider for X" formats.
- Run them with web search enabled, and record whether you are cited, how you are described, and which competitors appear in your place. That last column is usually the most instructive.
- Repeat monthly and correlate with changes you made. Technical fixes such as
robots.txtand schema tend to show up faster than authority building, which operates over months rather than weeks. - Fix the description, not just the presence. Being cited inaccurately usually traces back to inconsistent descriptions of your company across the web rather than to anything on your own site.
- Test in more than one language. Citation visibility is not uniform across languages, and a brand well cited in English can be absent in the markets where it actually sells.
That last point connects to everything else. Answer engines assemble responses from the sources they can find in the language of the question, so a company with strong English content and nothing in Bahasa Indonesia or Arabic is invisible at exactly the moment a customer in that market asks. Visibility, like the models themselves, is multilingual or it is partial.
See How ChatGPT picks sources and How Perplexity's answer engine picks sources for the equivalent mechanics on the other major surfaces.
Sources and further reading
- Anthropic Help Centre, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" — the three-crawler documentation.
- TechCrunch, on Brave Search powering Claude's web search.
- Erlin, "Claude SEO: How to Get Cited by Claude AI".
- AIDev, "The 2026 GEO Playbook".
- Pipeline Velocity, "Claude SEO: How To Get Your Site Cited In Claude Answers".