llms.txt explained: what it is, who reads it, and how to write one

Direct answer

llms.txt is a proposed Markdown file, hosted at a site's root, that lists the pages an AI agent should read first. It borrows the idea from robots.txt: a plain text file with a fixed shape that any crawler can find at a predictable URL. The short version of "who reads it": as of today, no major LLM provider has committed to reading it at all. Treat it as one small piece of answer engine optimization, not a ranking lever.

What it is and why it matters

HTML is built for people. A typical page wraps its content in navigation, ads and JavaScript, and pulling clean text back out of that is slow and imprecise for a language model working through many pages in a context window. The llms.txt specification exists to give an agent a shortcut: a curated, human-and-machine-readable index of a site's most important content, in a format language models already parse well, Markdown.

Jeremy Howard of Answer.AI wrote the spec, published September 3, 2024, and it's still evolving. The current v2 was last revised August 10, 2026. The required structure stays deliberately small: an H1 with the site or project name, the only mandatory element, a blockquote with a one-line summary, optional prose sections for context, and then one or more H2 sections that each hold a list of links.

That minimalism is the point. It isn't a full sitemap, and it isn't meant to replace one. A sitemap lists every URL for a crawler to index; llms.txt lists a much smaller, hand-picked set for an agent to actually read and reason over. Think of it as a table of contents an agent can skim in a single pass, then decide what to fetch next, rather than a complete index of everything a site publishes.

The name is a little misleading, too. It reads like a rule file, something in the shape of robots.txt that blocks or allows access. It isn't. Nothing in the spec grants or denies permission. An agent that ignores llms.txt entirely and crawls the rendered HTML instead hasn't violated anything, because there's no enforcement mechanism and no governing body behind the format yet, only a public specification and a community that has converged around it.

Kallos Labs treats llms.txt as one line item in a broader technical SEO audit, alongside sitemap coverage, schema and page speed, rather than as a standalone project. The file rarely changes the outcome of that audit on its own, but it's cheap enough that skipping it has no upside either.

How it works in practice

Writing an llms.txt file means following the format exactly, because agents parsing many sites benefit from consistency more than creativity. A minimal file opens with an H1 for the project name and a one-line blockquote summary, then organizes links under H2 sections like this:

## Docs

- [Getting Started](https://example.com/docs/start): Setup and first steps
- [API Reference](https://example.com/docs/api): Full endpoint list

Each entry in a link section is a standard Markdown link, optionally followed by a colon and a short description.

Serve the file as text/plain; charset=utf-8, per Vercel's implementation guide, so an agent can read the raw Markdown without depending on content type sniffing. For a Next.js site, that's a route handler at /llms.txt returning the string with that header set explicitly, the same pattern used for any other machine-readable endpoint, an RSS feed or a JSON sitemap included.

A few practical rules keep the file useful instead of decorative. Link out to canonical pages rather than pasting their content inline. The file is an index, not a mirror, and a stale copy inside llms.txt is worse than no copy at all once the source page changes. Keep the description after each link short and factual: a single clause is usually enough, since the point is helping an agent decide what to fetch, not selling the page. And group links under H2 headings that mean something, Docs, Pricing, Changelog, rather than one flat list, because the heading is the only signal an agent gets before deciding whether a section is relevant.

Two files can coexist: /llms.txt as a lean index that links out to canonical pages, and /llms-full.txt as a single file containing the complete text of everything referenced. That pairing earns its keep once a documentation set gets large enough that linking alone isn't enough context. The file can also live at a subpath, such as /docs/llms.txt, scoped to URLs beneath that path. When more than one applicable file exists, the more specific one wins, so a docs subdomain can maintain its own file without conflicting with the one at the root.

Tradeoffs and edge cases

Here's the honest state of adoption. Research covering 137,000 domains found that 28 percent, about 38,360 sites, publish a valid llms.txt file. But 97 percent of those files logged zero requests in May 2026, according to Ahrefs' analysis. Publishing one is not the same as being read.

There's a real signal underneath that number, though not a crawling commitment. Google referenced llms.txt inside its Agent2Agent protocol in April 2025, and in May 2026 it published an AI-optimization guide alongside experimental Lighthouse audits covering similar ground. OpenAI, Anthropic and Meta have made no equivalent statement.

The closest precedent is instructive. robots.txt was proposed in February 1994 and became a de facto standard within months, but it wasn't formally standardized until RFC 9309, in September 2022, twenty-eight years later. Compliance with it was, and still is, entirely voluntary; a crawler that ignores robots.txt faces no technical barrier, only a reputational one. llms.txt sits earlier in that same arc: informal, unenforced, and adopted faster than it's been confirmed useful.

That argues for a specific, modest recommendation. An llms.txt file costs little to add: one route, one Markdown file, a handful of links to a site's own best pages. It's worth shipping as part of a normal technical pass, alongside sitemap and schema work, the kind of AEO checklist item Kallos Labs runs for clients. It is not worth treating as a growth channel, a ranking factor, or a reason to delay the parts of technical SEO that already have measurable effect: page speed, structured data and indexation chief among them.

The clearer failure mode runs the other direction: teams that spend real engineering time building and maintaining an elaborate llms.txt, complete with automated regeneration and a full mirrored corpus in llms-full.txt, before they've fixed a slow Core Web Vitals score or a sitemap that omits half the site. Sequence matters here. Ship the file, then move on to the audit items with an actual track record.

Frequently asked questions

Does ChatGPT or Google actually read llms.txt?

No major LLM provider, OpenAI, Anthropic, Google or Meta, has formally committed to fetching or parsing llms.txt as of this writing. Google referenced the format inside its Agent2Agent protocol in April 2025, but a reference inside a protocol spec is not a crawling commitment.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may not access; llms.txt is a curated map of what a site wants an agent to read first. robots.txt took until RFC 9309 in September 2022, twenty-eight years after its 1994 debut, to become a formal standard, and llms.txt is still an informal proposal by comparison.

Where does llms.txt live and what format is it?

It lives at the site root, /llms.txt, served as text/plain; charset=utf-8. The file itself is Markdown: an H1 with the project name, a blockquote summary, and H2 sections listing links as [title](url): description.

What is llms-full.txt?

A companion file some sites publish alongside a lean /llms.txt index. Where llms.txt links out to canonical pages, llms-full.txt inlines the complete text of the referenced documentation in one file, which suits large documentation sets better than a link list alone.

Should a site publish an llms.txt file anyway?

Yes, as a low-cost addition, not a priority project. Research tracking 137,000 domains found 97 percent of published llms.txt files logged zero requests in May 2026, so it's a small bet on a future standard rather than a lever with a measurable return today.