Short answer. The technical work that decides AI search visibility is the same work that decides ordinary search visibility: crawlable URLs, main content present in the served HTML, correct status codes, self-consistent canonicals, real internal links, a sitemap that matches reality, and pages that render without requiring a browser. Generative features are built on the same crawling and indexing systems as ranked search, so a page that cannot be fetched and parsed is unavailable to both. No amount of answer-first rewriting fixes a page an engine never received in full — and this is the failure that is invisible from a browser, because in a browser everything looks fine.
Content teams are usually the ones asked why a page is not cited, and the answer is frequently outside their control. This is the checklist that separates a content problem from a delivery problem, ordered by how often each item is the actual cause.
Why does crawlability come before anything else?
Because retrieval is a filter with a hard edge. A page that is blocked, orphaned, noindexed, returning the wrong status code, or hidden behind a rendering path a fetcher cannot follow never enters the candidate pool. Nothing downstream of that — evidence density, passage structure, schema — has an effect on a page that was never a candidate.
The order of checks matters, because a failure at an early step makes later diagnostics meaningless:
robots.txt— is the path allowed, for every agent that matters, not only for a wildcard?- Robots meta and
X-Robots-Tag— is the pagenoindexby accident? Staging directives shipped to production are a routine cause. - Status codes — 200 for content, single-hop 301 for moves, and no soft 404s returning 200 with an error page.
- Canonical — self-referential on the canonical URL, and consistent with what the sitemap and internal links point at.
- Discoverability — is the page linked from another crawlable page, or reachable only through a search box or a JavaScript filter?
Only after all five pass is a content diagnosis worth running.
What should JavaScript sites verify?
This is the single most common cause of a good page being invisible, and it is the hardest to see, because the person checking is checking in a browser that executes JavaScript.
The test. Load the page with JavaScript disabled, or fetch the raw HTML and read it. What remains is approximately what a fetcher that does not execute JavaScript receives.
What to look for in that raw HTML:
- The main content, in full. Not a shell, not a loading state, not a placeholder.
- Real anchors with
hrefattributes. Navigation built from click handlers is invisible; the link graph that carries authority to deep pages has to exist in the markup. - Title, canonical and meta directives. Injected at runtime, these can arrive missing or wrong.
- Numbers that are actually numbers. Counters that animate up from zero render as
0in the served HTML. A crawler reading "0+ languages" is reading a stated fact about your company. - Content behind tabs and accordions. Collapsed is fine; conditionally rendered is not. A component that mounts a panel's contents only when opened serves one of several answers to a crawler and all of them to a human.
Where rendering is required to see the content, server-side rendering or pre-rendering is the fix. It is not a visibility guarantee — relevance, evidence and competition still decide the outcome — but it removes a class of failure that no content work can compensate for.
Are the AI agents being served the same page you are?
A page can pass every check above for a browser and still arrive truncated at a specific user agent. Rate limiters, bot-management rules and CDN defaults commonly return a shorter page, a challenge page, or a 403 to agents they do not recognise — and none of that is detectable from inside the site.
The verification is a byte-count comparison:
Delivery check = Bytes served to agent X ÷ Bytes served to a browser fetch
Run it per relevant user agent against a handful of important URLs. Anything materially below 1.0 is a delivery defect, not a content one. Two further points worth knowing:
- Training crawlers and retrieval crawlers are different bots doing different jobs, and common defaults block them under one toggle. Blocking a retrieval fetcher makes citation impossible.
- A managed bot rule you did not configure is still your rule. Platform and CDN defaults change, and they change without a deploy on your side, so this check belongs on a schedule rather than in a launch checklist.
How do internal links support AI-search discovery?
Internal links do two jobs: they make deep pages reachable, and their anchor text describes what the destination resolves.
| Pattern | Effect |
|---|---|
| Pillar page linking to focused pages, each linking back | Topic structure exists in the site, not only in the editorial plan |
| Anchor text stating the question the destination answers | The link describes the destination's purpose |
| Pages reachable only via search or a filtered list | Effectively orphaned for anything that does not execute the filter |
| Links generated client-side | Absent from the served HTML |
| Deep pages more than three clicks from an entry point | Crawled less, refreshed less |
A cluster that exists only as a content calendar is not a cluster. If the pages do not link to each other in the markup, the relationship is invisible to everything except the person who planned it.
Do page speed and Core Web Vitals matter here?
They matter for users and for ordinary search quality, and they are not a citation switch. There is no evidence that a faster page is more likely to be quoted, and treating performance work as AEO work misallocates effort.
The version of this that does matter: performance problems and visibility problems have a shared cause. Heavy client-side rendering slows the page and removes content from the served HTML. Layout that injects the answer late hurts both. Optimising images, scripts and fonts is worth doing on its own merits, and stripping useful content to hit a score is not.
The one performance-adjacent item with a direct effect is crawl efficiency: a site that times out under a fetcher's rate gets less of itself fetched.
What should a technical AI-search audit include?
Run these as a set, on templates rather than on individual pages, because one template defect affects every page built from it:
- Index coverage and the reasons for exclusion
- Served-HTML completeness on each page template
- Byte-count parity across relevant user agents
- Canonical consistency between page, sitemap and internal links
- Redirect chains reduced to single hops
- Status codes, including soft 404s
- Internal-link depth and orphan detection
- Sitemap accuracy — canonical, indexable URLs only, with real
lastmodvalues - Structured data that matches the visible page and does not contradict it
- Titles and descriptions present in the served HTML
- Publication and modification dates that reflect actual changes
Fix template-level defects before rewriting articles. One broken template is cheaper to fix than forty rewritten pages, and it is more often the cause.
How Lifewood approaches this
Lifewood treats delivery as a gate before content, on client sites and its own. The sequence is fixed: entity resolution, then machine readability, then publishing. Pages are checked as served rather than as rendered — JavaScript disabled, byte counts compared per agent, tab and accordion contents confirmed present in the markup — because the failure this catches is the one that silently invalidates every content decision downstream.
The reason the order is non-negotiable is measurement. Publishing into a delivery defect produces no movement and no way to tell whether the content was the problem. Baseline, fix delivery, then publish, then re-measure — otherwise the result is unattributable.
See AEO services and GEO services.
Sources and further reading
- Google Search Central, "Optimizing your website for generative AI features", "JavaScript SEO Basics" and "Search Essentials" — the official position that generative features depend on core Search technical requirements.
- Google Search Central, General Structured Data Guidelines — on markup matching the visible page.
- Companion guides: AI Crawlers and AI Search Visibility (training versus retrieval agents) and What Actually Gets You Cited by AI Answer Engines.