Short answer. llms.txt is a Markdown file placed at the root of a website that lists its most important pages for AI systems to read. It is a community proposal, not a standard, and the evidence that AI search systems actually read it is close to nonexistent: one study of 137,210 domains found 97% of published files received zero requests. It is cheap to ship and has one real, evidenced use case — developer documentation for coding agents — but it is not an AI visibility lever.
Key takeaways
llms.txtis a proposed convention from September 2024, not a ratified web standard, and no crawler is obligated to read it.- A study of 137,210 domains found 97% of published
llms.txtfiles received zero requests from any bot. - Adoption has grown sharply over the past year while actual usage stayed close to zero — the two curves point in opposite directions.
- Google has stated directly that Search does not use
llms.txt, and that no new machine-readable files are needed to appear in AI-generated results. - The one substantiated use case is developer documentation: AI coding agents fetch
llms.txtfiles more than any other type of AI retrieval bot.
What is llms.txt?
A Markdown file at yoursite.com/llms.txt that points AI systems at your best content, proposed in September 2024 by Jeremy Howard of Answer.AI.
The idea is straightforward. Web pages are cluttered with navigation, scripts and boilerplate, and a language model working within a limited context window wastes tokens parsing all of it. llms.txt is a clean, structured summary of a site with links to the pages that matter, in a format a model can read cheaply — the same logic behind structuring a site so AI engines can cite it more broadly.
Two things it is not, both widely misunderstood:
- It is not a blocking tool.
llms.txtcannot restrict any crawler or prevent any AI system from reading your site.robots.txtis the file that controls crawler access, and confusing the two leaves people believing they have controlled access when they have not — the distinction matters for anyone deciding which AI crawlers to allow or block. - It is not a standard. There is no backing from the W3C, the IETF or any recognised standards body, and no enforcement mechanism. AI providers adopt it, or ignore it, entirely on their own terms.
Does anything actually read it?
Overwhelmingly, no. The adoption curve and the usage curve point in opposite directions.
| Measure | Finding |
|---|---|
| Adoption rate | 10.13% of ~300,000 domains analysed |
| Files receiving requests | 97% received zero requests across all 137,210 domains studied |
| Largest source of traffic to these files | SEO and tech-profiling audit tools checking whether a site has one, not AI retrieval bots |
| Correlation with AI citation frequency | No statistically significant correlation found |
Ahrefs examined every domain in its analytics that received traffic in May 2026, checked each root for a live llms.txt, and then measured every request to those paths. Their summary was blunt: AI retrieval bots barely fetch these files, and no AI system goes looking for one a site has not published.
Two details make the picture stranger. Part of the llms.txt economy is tools measuring compliance with a convention almost nobody consumes. And separate monitoring of over 500 million AI bot events found only a few hundred requests targeting llms.txt directly, with the major retrieval crawlers skipping it and fetching HTML instead — a reminder that how AI search engines actually decide what to mention or cite still runs mostly through ordinary page content.
Why do people disagree about it?
Because the platforms are sending mixed signals, and because one genuine use case is being generalised into a claim about search visibility.
Google has been the most direct. Gary Illyes said in July 2025 that Google does not support llms.txt and is not planning to, and John Mueller compared it to the discredited keywords meta tag. Google's AI optimisation documentation, updated 15 June 2026, states that a site does not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search including its generative capabilities, because Search does not use them.
Mueller also gave the structural argument, which is the most persuasive one against the file as a ranking signal: a self-reported manifest cannot differentiate between sites, because every site would claim to be the best one. A signal a site writes about itself is not an independent signal — the same reason structured data and entity identity carry more weight when they can be checked against outside sources.
Yet Google added an llms.txt audit to Lighthouse in May 2026, filed under a new agentic browsing category. That contradiction fuels much of the confusion, and it points at where the file actually has value: agent-readiness infrastructure, meaning tooling built for AI coding agents rather than for search or answer engines. Companies including Stripe, Cloudflare and Anthropic publish one, and they are all documentation-heavy. In the Ahrefs dataset, the agent that fetched llms.txt more than any AI retrieval bot was a coding agent, which fits the proposal's docs-first origins.
So should you ship one?
Probably yes if a site has substantial documentation, probably not as a priority otherwise, and never instead of the things that actually move citations.
Ship one if a site publishes developer documentation, an API reference or a large technical knowledge base that AI coding agents might consult — the benefit there is concrete and immediate. Deprioritise it for a marketing site, a services business or a publisher hoping for more AI citations, since the evidence does not support that use and the file will most likely sit unread.
Two failure modes are worth avoiding beyond simply skipping it. Generating a Markdown copy of every page creates duplicate content at scale, which dilutes crawl budget and can suppress rankings for the originals. And a file that goes stale is worse than none, because it publishes a self-description that no longer matches the site.
Measuring rather than assuming settles the question for any specific site: filter access logs for requests to /llms.txt by known AI user agents, or place a unique URL inside the file that appears nowhere else and watch whether anything follows it. That gives a factual answer within weeks.
The opportunity cost is the real argument. Every hour spent polishing a document the machines skip is an hour not spent on crawler configuration, answer-shaped content structure, original data and third-party corroboration — all of which have measurable effects. That is also the ordering behind generative engine optimization as a discipline: access first, then structure, then evidence, with emerging conventions like llms.txt treated as cheap housekeeping rather than strategy. Lifewood's own answer engine optimization work follows the same sequence for clients weighing where to spend engineering time.