llms.txt is a proposed convention: a Markdown file at your domain root that describes your site to language models in a form they can consume cheaply.
It is worth understanding, because it comes up constantly and the honest assessment — good idea, essentially no adoption — is not what most articles about it say.
The problem it addresses
A model trying to understand your site has to fetch HTML and work through navigation, cookie banners, scripts and boilerplate to reach the content. This is wasteful and lossy, particularly for documentation, where the structure carries real meaning that markup flattens.
The proposal: publish /llms.txt, a Markdown document that says what the site is and links to the pages that matter, optionally with one-line descriptions. Some sites also publish /llms-full.txt containing the full content of key pages inline.
Think of it as a sitemap written for reading rather than for crawling.
What it looks like
# GhostPane
> Cookieless, self-hosted web analytics that runs inside your own
> Cloudflare account and can measure from the request path.
## Documentation
- [Getting started](https://example.com/docs/start): Install and first site
- [Edge mode](https://example.com/docs/edge): Counting from the request path
- [Stats API](https://example.com/docs/api): Read-only bearer-token access
## Background
- [How cookieless counting works](https://example.com/blog/how-cookieless-analytics-counts-visitors)
## Optional
- [Changelog](https://example.com/changelog)A title, a summary in a blockquote, then linked sections. ## Optional is a defined convention meaning "skip this if you are short on context".
The part nobody wants to say
No major model provider has committed to reading it. Not OpenAI, not Anthropic, not Google. There is no announced support, and no reliable evidence of it changing how any assistant treats a site.
It is a proposal with genuine merit that has been adopted by publishers and not by consumers. That is a normal stage for a standard, and it is also the stage many standards never leave.
So the accurate framing is: this is a cheap bet on a plausible future, not a technique with a measurable return today. Anyone telling you llms.txt improves your AI visibility right now is describing a hope.
Should you publish one?
Yes, if you run documentation or a developer tool, the file is easy to generate from structure you already have, and being the source an assistant answers from is worth something to you. The cost is close to zero and the option value is real.
Not a priority if you are a general content site. Your effort is better spent on things that demonstrably work: clear structure, real headings, being genuinely the best answer, and understanding which crawlers already read you.
Do not treat it as a substitute for robots.txt. They do unrelated jobs — robots.txt controls access and is honoured, llms.txt describes content and is mostly not.
If you publish one
Keep it current. A stale llms.txt is worse than none, because it confidently describes a site that no longer exists. Generate it from the same source as your navigation so it cannot drift.
Keep it short. The point is to save context, so a file listing four hundred URLs defeats itself. Link the twenty pages that matter.
Do not duplicate your sitemap. A sitemap is exhaustive and machine-oriented; llms.txt is curated and meaning-oriented. If yours is just the sitemap in Markdown, it is adding nothing.
How you would know if it worked
You would see it fetched. That is the whole test, and it is the same measurement as everything else in this area: requests for /llms.txt in your logs, by agent, over time.
If nobody is fetching it, it is doing nothing — which, today, is the likely finding. That is fine. It costs almost nothing to have published, and the day something does start reading it, you are already there.
Common questions
Do any AI models actually read llms.txt?
No major provider has committed to it. OpenAI, Anthropic and Google have not announced support, and there is no reliable evidence that publishing the file changes how any assistant treats a site. It is a well-designed proposal that publishers have adopted and consumers have not — worth doing as a cheap bet, not as a technique with a measurable return today.
Is llms.txt a replacement for robots.txt?
No, they do unrelated jobs. robots.txt controls access and is honoured by every major crawler; llms.txt describes your content and is currently honoured by essentially nobody. If you want to influence what AI crawlers can fetch, that is entirely a robots.txt matter, and llms.txt has no bearing on it.
How do I know whether my llms.txt is being used?
Watch for requests to the file itself in your server or edge logs, broken down by user agent. It is a single URL, so it is easy to track. If nothing is fetching it, it is having no effect — which is the likely finding today, and is a reason to keep it cheap rather than a reason not to have one.