What Is llms.txt, and Does Your Website Need One?

Short answer. llms.txt is a Markdown file placed at the root of a website that lists its most important pages for AI systems to read. It is a community proposal, not a standard, and the evidence that AI search systems actually read it is close to nonexistent: one study of 137,210 domains found 97% of published files received zero requests. It is cheap to ship and has one real use case — developer documentation for coding agents — but it is not an AI visibility lever, and treating it as one costs the hours that would move citations.

Few conventions have generated as much activity on as little evidence. Adoption grew 8.8-fold in a year while measured usage stayed near zero, and the gap between those two curves is the whole story. This piece covers what the file is, what reads it, why the platforms disagree, and when shipping one is actually worth your time.

---

What is llms.txt?

A Markdown file at yoursite.com/llms.txt that points AI systems at your best content, proposed in September 2024 by Jeremy Howard of Answer.AI.

The idea is straightforward. Web pages are cluttered with navigation, scripts and boilerplate, and a language model working within a limited context window wastes tokens parsing all of it. An llms.txt file offers a clean, structured summary of the site and links to the pages that matter, in a format a model can read cheaply.

Two things it is not, both widely misunderstood:

• It is not a blocking tool. llms.txt cannot restrict any crawler or prevent any AI system from reading your site. That is what robots.txt is for, and confusing the two leaves people believing they have controlled access when they have not.
• It is not a standard. There is no backing from the W3C, the IETF or any recognised standards body, and no enforcement mechanism. AI providers adopt it, or ignore it, entirely on their own terms.

---

Does anything actually read it?

Overwhelmingly, no. The adoption curve and the usage curve point in opposite directions.

Ahrefs examined every domain in its analytics that received traffic in May 2026, checked each root for a live llms.txt, and then measured every request to those paths. Their summary was blunt: AI retrieval bots barely fetch these files, and no AI system goes looking for one you have not published.

Two details make the picture stranger. Part of the llms.txt economy is tools measuring compliance with a convention almost nobody consumes. And separate monitoring of over 500 million AI bot events found only a few hundred requests targeting llms.txt directly, with the major retrieval crawlers skipping it and fetching HTML instead.

---

Why do people disagree about it?

Because the platforms are sending mixed signals, and because one genuine use case is being generalised into a claim about search visibility.

Google has been the most direct. Gary Illyes said in July 2025 that Google does not support llms.txt and is not planning to, and John Mueller compared it to the discredited keywords meta tag. Google's AI optimisation documentation, updated 15 June 2026, states that you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search including its generative capabilities, because Search does not use them.

Mueller also gave the structural argument, which is the most persuasive one against the file as a ranking signal: a self-reported manifest cannot differentiate between sites, because every site would claim to be the best one. A signal you write about yourself is not a signal.

Yet Google added an llms.txt audit to Lighthouse in May 2026, filed under a new agentic browsing category. That contradiction fuels much of the confusion — and it points at where the file actually has value.

The real use case is developer tooling. AI coding assistants retrieve documentation in real time, and a clean index saves them tokens and wrong turns. In the Ahrefs dataset, the agent that fetched llms.txt more than any AI retrieval bot was a coding agent, which fits the proposal's docs-first origins. Companies including Stripe, Cloudflare and Anthropic publish one, and they are all documentation-heavy.

The honest summary: llms.txt is agent-readiness infrastructure, not a search visibility tool. Both camps are describing something real and labelling it differently.

---

So should you ship one?

Probably yes if you have substantial documentation, probably not as a priority otherwise, and never instead of the things that actually move citations.

Ship one if you publish developer documentation, an API reference or a large technical knowledge base that AI coding agents might consult. The benefit is concrete and immediate.

Deprioritise it if you are a marketing site, a services business or a publisher hoping for more AI citations. The evidence does not support that use, and the file will most likely sit unread.

Do not ship a bad one. Two failure modes are worth avoiding. Generating a Markdown copy of every page creates duplicate content at scale, which dilutes crawl budget and can suppress rankings for the originals. And a file that goes stale is worse than none, because you have published a self-description that no longer matches your site.

Measure rather than assume. Filter your access logs for requests to /llms.txt by known AI user agents, or place a unique URL inside the file that appears nowhere else and watch whether anything follows it. You will have a factual answer within weeks.

The opportunity cost is the real argument. Every hour spent polishing a document the machines skip is an hour not spent on crawler configuration, answer-shaped content structure, original data and third-party corroboration — all of which have measurable effects. That ordering is where Lifewood's AEO and GEO work starts: access first, then structure, then evidence, with emerging conventions treated as cheap housekeeping rather than strategy.

---

Sources and further reading

• Ahrefs, "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read" — the zero-request finding and the coding-agent observation.
• PPC Land, on the Originality.ai tracking study and 8.8× adoption growth.
• Digital Applied, "llms.txt in Practice: Adoption Data, Evidence, and Setup" — Google's June 2026 documentation and the Lighthouse audit.
• Ecorpit, "Does llms.txt help SEO? Google's 2026 answer".
• Derivatex, "LLMs.txt Guide: What It Does and Doesn't Do".