What is llms.txt, and is it worth publishing?

Updated Published 7 min read

In short

llms.txt is a proposed convention for a markdown file at a site's root that summarises what the site covers and links to its most important pages, each with a short description. It is not a standard, and no major answer engine has confirmed reading it as a retrieval or ranking input. It costs very little to publish, which is the honest case for doing so.

Key points

  • llms.txt is a proposed convention published at llmstxt.org, not a W3C or IETF standard, and adoption by answer engines is unconfirmed.
  • The format is markdown: an H1 site name, a blockquote summary, then H2 sections of links written as - [Title](url): description.
  • It differs from a sitemap by being curated and annotated — an llms.txt that lists every URL adds nothing over the sitemap it came from.
  • The Optional section is defined by the convention as material a reader may skip, which is where legal and administrative pages belong.
  • A stale llms.txt is worse than none, because it describes a site that no longer exists with apparent authority — generate it from real content rather than maintaining it by hand.

If you have seen llms.txt mentioned and want a straight answer on whether to spend an afternoon on one, here it is: the format takes twenty minutes to understand, the file takes an hour to write well, and nobody can currently demonstrate that any major answer engine reads it.

That is not a reason to skip it. It is a reason to size the effort accordingly, and to be sceptical of anyone selling it as a ranking lever.

What it is

llms.txt is a proposed convention for a plain-text, markdown-formatted file at the root of a site — https://example.com/llms.txt — that describes what the site covers and points at its most important pages.

The reasoning behind it is straightforward. A language model answering a question about your site has a limited context budget. Crawling forty pages to work out what you do is expensive and lossy. A single curated file that says "this is what we are, and here are the twelve pages that matter, and here is what each one is for" is cheap to read and hard to misinterpret.

The comparison people reach for is robots.txt, and it is a decent one as far as placement and format go. It breaks down on authority: robots.txt is a long-established convention that every major crawler honours, and llms.txt is a proposal from 2024 that nobody has committed to.

The format

# Site name

> One sentence describing what this site is for.

## Docs

- [Getting started](https://example.com/docs/start): Install and first run.
- [API reference](https://example.com/docs/api): Every endpoint, with examples.

## Guides

- [Migrating from v1](https://example.com/guides/migrate): Breaking changes and the upgrade path.

## Optional

- [Privacy policy](https://example.com/privacy): What we collect.

Four elements, in order:

  • An H1 with the site or project name. Exactly one.
  • A blockquote summary. The most valuable line in the file, because it is the first thing read and the thing most likely to be used verbatim. Write it as though it were the only line that survived.
  • H2 sections of links, each entry - [Title](url): description. The description after the colon is the point of the whole exercise.
  • An Optional section, which the convention defines specifically as material that can be skipped when context is short. Legal pages, terms, administrative pages.

There is a companion file, llms-full.txt, that contains the full text of those pages inline rather than links to them. It saves a model from fetching each page separately. It suits documentation and reference material; for a large content site it becomes unwieldy quickly, and the curated link file is the more useful of the two.

How it differs from a sitemap

This is where most implementations go wrong, so it is worth being blunt.

A sitemap is exhaustive and unannotated. It exists for a crawler that intends to fetch every URL anyway, and its job is completeness — every page, with a last-modified date.

An llms.txt is selective and annotated. It exists for a reader that will not fetch everything, and its job is judgement: which twenty pages represent this site, and what is each one for.

If your llms.txt is your sitemap with different punctuation, it adds nothing. You have produced a longer file with less metadata than the XML you already had. The value is entirely in the descriptions and the leaving-out.

Twenty well-described links beat four hundred bare ones. The test for including a page: would it make a hand-written "start here" list you gave a new colleague?

What the evidence actually is

Here is the part usually left out.

As of this writing, no major answer engine has publicly confirmed that it reads llms.txt as a retrieval or ranking input. There is no announcement from OpenAI, Anthropic, Google or Perplexity committing to it. What exists is a well-argued proposal, meaningful adoption among documentation sites and developer tools, and a fair amount of writing that treats adoption by publishers as though it were adoption by engines. Those are different things.

So the honest case for publishing one is narrow and it is not about rankings:

  • It costs very little. An hour to write, or minutes if generated from content you already have.
  • The downside is bounded. The worst case is a file nothing reads.
  • If adoption arrives, you already have a curated index rather than whatever a crawler infers from your navigation — and you will not be writing it under time pressure.
  • Writing it is clarifying. Being forced to name the twelve pages that represent your site, and say what each is for in one line, tends to expose that you cannot.

That is the whole argument. Treat any tool, plugin or agency promising traffic from an llms.txt with the suspicion that claim deserves — and note that this site publishes one, generates one from its plugin, and still will not tell you it moves a number.

Publishing it

The file has to serve as plain text from the root of the domain: https://yoursite.com/llms.txt. A few practical notes:

  • On WordPress, uploading it to the media library serves it from /wp-content/uploads/, which is not the root and will not be found. It needs a real file at the web root or a route registered by a plugin or your theme.
  • Serve it as text/plain. Some hosts will guess text/markdown, which is harmless, but an HTML error page returned with a 200 is not.
  • Confirm a crawler can reach it. A summary file behind a blocked retrieval crawler is effort spent on something nothing is permitted to read — see training vs. retrieval crawlers.

Our llms.txt generator will take a sitemap or a list of URLs and build the scaffolding, leaving you the descriptions — which is the part worth your time and the part a generator cannot do for you.

Keeping it true

A stale llms.txt is worse than not having one. It describes a site that no longer exists, in a confident voice, in a file whose entire premise is that it can be trusted without verification. Sections get renamed, pages get retired, and the summary quietly becomes a description of last year.

The durable fix is to generate it from real content rather than maintaining it by hand — which is what this site does for its own file, and what the Citely plugin does for a WordPress site. If you are writing yours by hand, put a recurring reminder somewhere to re-read it.

So: should you publish one?

Yes, if it takes you an hour or less and you will keep it current. It is cheap insurance and the writing exercise has its own value.

No, if it means a week of work, or if you would be doing it instead of making your pages quotable. The properties that decide whether a passage gets lifted have actual research behind them. This one has a proposal and a hope, and it should be priced that way.

Sources

Technical claims in this guide are drawn from the primary documentation and specifications below, each confirmed live on the date shown.

  1. The /llms.txt file proposal llmstxt.org. Accessed .
  2. Sitemaps XML format sitemaps.org. Accessed .

FAQ

Is llms.txt an official standard?
No. It is a proposed convention published at llmstxt.org in 2024. It has not been adopted by a standards body, and no major answer engine has publicly committed to reading it. Comparisons to robots.txt describe the format and placement accurately but overstate the authority.
Does llms.txt improve my rankings or AI visibility?
There is no public evidence that it does. No major answer engine has confirmed using it as a retrieval or ranking input. The reasonable case for publishing one is that it costs very little and positions you if adoption arrives — not that it will move traffic. Anyone claiming a measured ranking effect should be asked what they measured it against.
How is llms.txt different from a sitemap?
A sitemap is exhaustive and unannotated, written for a crawler that will fetch every URL anyway. An llms.txt is curated and annotated, written for a reader that will not fetch everything: it names the pages that matter and says what each is for. An llms.txt that simply lists every URL adds nothing over the sitemap it came from.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a curated list of links with one-line descriptions. llms-full.txt contains the full text of those pages inline, so a model does not need to fetch each one separately. The full version suits documentation and reference sites; on a large content site it becomes very large, and the curated link file is generally the more useful of the two.
Where does the llms.txt file go on WordPress?
It must serve from the root of the domain at /llms.txt. Uploading it through the media library puts it under /wp-content/uploads/, which will not be found. You need either a real file at the web root or a plugin or theme that registers a route serving it as plain text.
How many links should an llms.txt contain?
As few as represent the site honestly — typically ten to thirty for most sites. The test is whether a page would appear on a hand-written 'start here' list for a new colleague. Twenty links with good descriptions are more useful than four hundred bare ones, and the leaving-out is as much of the value as the including.

Check any URL against everything in this guide — crawler access, extractability, attribution and structured data — in a few seconds.

Run the free checker
Go multilingual free
Open source · your data stays yours
Start free