If you have seen llms.txt mentioned and want a straight answer on whether to spend an afternoon on one, here it is: the format takes twenty minutes to understand, the file takes an hour to write well, and nobody can currently demonstrate that any major answer engine reads it.
That is not a reason to skip it. It is a reason to size the effort accordingly, and to be sceptical of anyone selling it as a ranking lever.
What it is
llms.txt is a proposed convention for a plain-text, markdown-formatted file at the root of a site — https://example.com/llms.txt — that describes what the site covers and points at its most important pages.
The reasoning behind it is straightforward. A language model answering a question about your site has a limited context budget. Crawling forty pages to work out what you do is expensive and lossy. A single curated file that says "this is what we are, and here are the twelve pages that matter, and here is what each one is for" is cheap to read and hard to misinterpret.
The comparison people reach for is robots.txt, and it is a decent one as far as placement and format go. It breaks down on authority: robots.txt is a long-established convention that every major crawler honours, and llms.txt is a proposal from 2024 that nobody has committed to.
The format
# Site name
> One sentence describing what this site is for.
## Docs
- [Getting started](https://example.com/docs/start): Install and first run.
- [API reference](https://example.com/docs/api): Every endpoint, with examples.
## Guides
- [Migrating from v1](https://example.com/guides/migrate): Breaking changes and the upgrade path.
## Optional
- [Privacy policy](https://example.com/privacy): What we collect.
Four elements, in order:
- An H1 with the site or project name. Exactly one.
- A blockquote summary. The most valuable line in the file, because it is the first thing read and the thing most likely to be used verbatim. Write it as though it were the only line that survived.
- H2 sections of links, each entry
- [Title](url): description. The description after the colon is the point of the whole exercise. - An
Optionalsection, which the convention defines specifically as material that can be skipped when context is short. Legal pages, terms, administrative pages.
There is a companion file, llms-full.txt, that contains the full text of those pages inline rather than links to them. It saves a model from fetching each page separately. It suits documentation and reference material; for a large content site it becomes unwieldy quickly, and the curated link file is the more useful of the two.
How it differs from a sitemap
This is where most implementations go wrong, so it is worth being blunt.
A sitemap is exhaustive and unannotated. It exists for a crawler that intends to fetch every URL anyway, and its job is completeness — every page, with a last-modified date.
An llms.txt is selective and annotated. It exists for a reader that will not fetch everything, and its job is judgement: which twenty pages represent this site, and what is each one for.
If your llms.txt is your sitemap with different punctuation, it adds nothing. You have produced a longer file with less metadata than the XML you already had. The value is entirely in the descriptions and the leaving-out.
Twenty well-described links beat four hundred bare ones. The test for including a page: would it make a hand-written "start here" list you gave a new colleague?
What the evidence actually is
Here is the part usually left out.
As of this writing, no major answer engine has publicly confirmed that it reads llms.txt as a retrieval or ranking input. There is no announcement from OpenAI, Anthropic, Google or Perplexity committing to it. What exists is a well-argued proposal, meaningful adoption among documentation sites and developer tools, and a fair amount of writing that treats adoption by publishers as though it were adoption by engines. Those are different things.
So the honest case for publishing one is narrow and it is not about rankings:
- It costs very little. An hour to write, or minutes if generated from content you already have.
- The downside is bounded. The worst case is a file nothing reads.
- If adoption arrives, you already have a curated index rather than whatever a crawler infers from your navigation — and you will not be writing it under time pressure.
- Writing it is clarifying. Being forced to name the twelve pages that represent your site, and say what each is for in one line, tends to expose that you cannot.
That is the whole argument. Treat any tool, plugin or agency promising traffic from an llms.txt with the suspicion that claim deserves — and note that this site publishes one, generates one from its plugin, and still will not tell you it moves a number.
Publishing it
The file has to serve as plain text from the root of the domain: https://yoursite.com/llms.txt. A few practical notes:
- On WordPress, uploading it to the media library serves it from
/wp-content/uploads/, which is not the root and will not be found. It needs a real file at the web root or a route registered by a plugin or your theme. - Serve it as
text/plain. Some hosts will guesstext/markdown, which is harmless, but an HTML error page returned with a 200 is not. - Confirm a crawler can reach it. A summary file behind a blocked retrieval crawler is effort spent on something nothing is permitted to read — see training vs. retrieval crawlers.
Our llms.txt generator will take a sitemap or a list of URLs and build the scaffolding, leaving you the descriptions — which is the part worth your time and the part a generator cannot do for you.
Keeping it true
A stale llms.txt is worse than not having one. It describes a site that no longer exists, in a confident voice, in a file whose entire premise is that it can be trusted without verification. Sections get renamed, pages get retired, and the summary quietly becomes a description of last year.
The durable fix is to generate it from real content rather than maintaining it by hand — which is what this site does for its own file, and what the Citely plugin does for a WordPress site. If you are writing yours by hand, put a recurring reminder somewhere to re-read it.
So: should you publish one?
Yes, if it takes you an hour or less and you will keep it current. It is cheap insurance and the writing exercise has its own value.
No, if it means a week of work, or if you would be doing it instead of making your pages quotable. The properties that decide whether a passage gets lifted have actual research behind them. This one has a proposal and a hope, and it should be priced that way.