llms.txt
A plain-text file at the root of your site describing what it is and what is worth reading. Crawlune publishes one in a click.
Set your expectations first
llms.txt is a proposed convention, not
a standard, and adoption by AI companies is partial and changing.
Publishing one is cheap, harmless and plausibly useful. It is not
transformative, and anyone telling you it is has something to sell
you.
Crawlune offers it because the cost is close to zero and the potential upside is real — not because it can promise you a result from it.
What Crawlune publishes
A summary of your site, then a curated list of the pages worth reading, with a line on each. What goes in is under your control: you choose which content types are eligible and can exclude anything individually.
It is generated from your actual published content, so it does not go stale in the way a hand-written file does.
It is not a permission system
An llms.txt file describes; it does not restrict.
Nothing in it stops a crawler reading anything. If your question is
"who is allowed to read my site", that is
crawler controls, which
is a different file and a different mechanism.
If you have not published one
Crawlune says so plainly on its screens rather than scoring you down silently for it. Publishing is one action, and you can see the file's contents before it goes live.