llms.txt: what it is, who actually reads it, and whether your site needs one

AUGUST 7, 2026 · 5 MIN READ · LLMS.TXT / AEO / TECHNICAL

llms.txt is a proposed convention: a plain markdown file served at /llms.txtthat gives large language models a short, curated map of your site’s most important content. Should you add one? Probably yes — it takes about twenty minutes and carries no downside — but go in with clear eyes: no major AI provider has publicly confirmed that its crawlers use the file. This is cheap insurance, not a ranking lever.

The proposal

The idea was put forward by Jeremy Howard of Answer.AI in late 2024. The observation behind it is sound: when an AI system lands on your site with a limited context window, it has to guess which of your hundreds of URLs matter, and it has to wade through navigation, cookie banners and boilerplate to find the substance. llms.txt proposes a fix in the spirit of robots.txt — a well-known location where you hand over the answer directly. The file is ordinary markdown: an H1 with the site name, a one-line summary in a blockquote, then sections of links with a short description of what each page contains. A companion convention, llms-full.txt, goes further and inlines full page content as one big markdown document.

How it differs from robots.txt and sitemap.xml

• robots.txt is access control: which crawlers may fetch which paths. It says nothing about what the content is.
• sitemap.xml is exhaustive inventory: every indexable URL, machine-readable, zero editorial judgment. Its job is completeness.
• llms.txt is curation: a human-written shortlist of the pages that actually explain your product, with context attached. Its job is judgment — the thing the other two files deliberately lack.

They are complementary, not competing. You still want all three; they answer different questions from different readers.

The honest state of adoption

Here is the part most write-ups mumble through. Plenty of sites now publishan llms.txt — documentation platforms generate them automatically, AI companies’ own docs sites carry them, and there are directories cataloguing thousands of examples. But publishing is the easy half. On the consuming side, OpenAI, Google and Anthropic have not stated that their crawlers or assistants read the file, and Google’s search representatives have been openly skeptical of it in public comments. Server logs around the web show occasional fetches of /llms.txt, which proves curiosity, not influence. Anyone telling you llms.txt is currently moving AI visibility numbers is ahead of the evidence.

Why it’s still worth twenty minutes

• The cost is near zero. One static markdown file, no build complexity, no risk to anything else on the site.
• The downside is zero. Unlike a robots.txt mistake, a bad llms.txt blocks nothing and breaks nothing. Worst case, it is ignored — which is also the current expected case.
• If adoption comes, you are already there. Conventions like this succeed or die on network effects. Being early costs nothing; retrofitting later is the same twenty minutes, just under time pressure.
• The exercise itself is useful. Forcing yourself to write one line about each page that matters is a miniature content audit. If you cannot summarize a page for the file, the page probably has a clarity problem an answer engine will also notice.

What to put in one

Keep it short and current. Start with your site name as the H1 and a one-sentence description of what the site is, in a blockquote — this is the single line a model is most likely to actually use. Then add a handful of sections — Docs, Product, Pricing, Blog, FAQ — each containing a few links, and give every link a plain one-line description of what a reader finds there. Prioritize the pages that answer buyer questions: what the product does, what it costs, how it compares. Skip the archive sprawl; the whole point is that this file is the shortlist, not the sitemap. And put a reminder somewhere to update it when your pages change — a confidently wrong llms.txt is worse than none.

We practice what we describe: hawkorbit.com serves its own /llms.txt, curated by hand. Whether anything reads it yet is exactly the kind of question you should be measuring rather than assuming — which is why we track AEO outcomes directly, by asking the engines. If you want to see what AI assistants currently say about your brand, the AEO Visibility Checker will ask them for you — llms.txt or not.