llms.txt is a markdown file you publish at yourdomain.com/llms.txt that gives AI systems a curated, human-written summary of your site: what it is, and links to the pages that matter most. It's a real, easy-to-ship file. It is not, despite what a lot of GEO content implies, something the major AI answer engines have committed to reading. A GEO agency that tells you otherwise is selling hype.
Key Takeaways
- llms.txt is a markdown file at /llms.txt that summarizes your site and links to key pages, for LLMs instead of search crawlers.
- It was proposed by Jeremy Howard (Answer.AI) in September 2024: it's a community convention, not a ratified web standard.
- Ahrefs analyzed 137,210 domains and found 97% of llms.txt files received zero requests in May 2026, and AI retrieval bots were just 1.1% of the traffic that did arrive.
- As of 2026, no major AI provider (OpenAI, Google, Anthropic, Meta, or Mistral) has publicly committed to reading it in production; Google's documentation says you don't need machine-readable AI files at all.
- SE Ranking tested ~300,000 domains and found no correlation between llms.txt and AI citations, removing it improved their prediction model.
- It's still worth shipping for developer-facing products: coding agents (Cursor, Claude Code, GitHub Copilot, Windsurf, Cline, Aider) do consume it, and it costs almost nothing to add.
- It is not a substitute for the things that actually move AI-answer inclusion: schema, entity clarity, and citable content.
Definition: llms.txt
llms.txt is a proposed convention, not a web standard, for a markdown file at the root of a website (/llms.txt) that gives large language models
a concise, curated overview of the site: what it is, in an H1 and a one-line blockquote summary, followed by H2-organized lists of links to the
documentation, product, and reference pages worth reading. An optional /llms-full.txt provides the same content inlined in full, for models that
can't or won't follow the links.
Where it came from
Jeremy Howard, founder of Answer.AI (and before that, fast.ai), published the llms.txt proposal in September 2024. The pitch was simple: LLMs have limited context windows and no reliable way to tell your marketing pages from your API reference from your changelog. A robots.txt tells crawlers what not to fetch; a sitemap.xml lists every URL with no sense of priority. Neither was built for a system trying to answer "how do I do X with this product" from a handful of pages it can actually fit in context.
llms.txt fills that gap for a narrow, real use case: developer tools, API docs, and technical products where an AI coding assistant benefits from a short, hand-picked reading list instead of crawling your whole site. Answer.AI, Anthropic's own documentation, Vercel, and Stripe all publish one.
What actually goes in the file
The format is deliberately minimal. No new schema to learn, just markdown:
# Your Project Name
> One-sentence summary of what this is and who it's for.
## Docs
- [Getting Started](/docs/start): Setup and first steps
- [API Reference](/docs/api): Full endpoint reference
## Optional
- [Changelog](/changelog): Recent updates
An H1 with the project name, a blockquote summary directly under it, then H2-labeled sections, typically "Docs," "Examples," "Optional", each a linked list with a short description of what's behind the link. That's the entire spec. If your content is short enough to inline, /llms-full.txt skips the links and pastes the actual content in one file.
The part most articles skip: does anyone read it?
This is where the data diverges from the hype. We now have server-log measurements instead of speculation.
Ahrefs analyzed 137,210 domains and found that 97% of llms.txt files received zero requests in May 2026: no bots, no humans, nothing at all (Ahrefs). Of roughly 38,000 domains with a valid file, only about 1,100 were fetched by anything. And when you look at what did the fetching, the picture gets worse for the AI-search theory:
AI retrieval bots (OAI-SearchBot and PerplexityBot, the crawlers that actually feed AI search answers) accounted for 1.1% of all requests. SEO audit tools generated twenty times more traffic to these files than the engines they were written for. Ahrefs put the most damning comparison plainly: "Slackbot alone fetched llms.txt files more often than PerplexityBot did."
Two more findings from that dataset matter strategically:
- Zero requests came from AI bots for llms.txt files that don't exist. They never go looking. Publishing one does not put you on any AI system's radar: the file only gets fetched when something already points to it.
- 96% of the requests that did arrive came from bots, and a chunk of the human traffic was SEOs sharing llms.txt links in chat apps, where link-preview bots dutifully fetched them.
Google has now said it in writing
The earlier evidence was a series of on-the-record remarks. Google's John Mueller compared the file to a long-dead ranking signal in April 2025:
"AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag: this is what a site-owner claims their site is about."
Gary Illyes followed at a July 2025 Search Central event, stating Google does not support llms.txt and is not planning to. Mueller later added the structural reason it can't work for discovery: an LLM "by design, can't trust what is here as a way of differentiating between different websites." A file you write about yourself is not evidence, it's a claim.
That position is now in Google's own documentation. Its guidance on AI features states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features" (Google Search Central). Worth being precise, since plenty of GEO content overstates this: that page does not name llms.txt. It rules out the whole category of machine-readable AI files, which includes llms.txt but isn't a targeted verdict on it.
Adoption is broad, shallow, and flat
Roughly one in ten sites publishes the file. SE Ranking found 10.13% adoption across ~300,000 domains: and, tellingly, the largest sites adopt it least (SE Ranking).
That flatness is the argument against FOMO. If llms.txt were quietly working, you would expect the biggest, best-resourced sites to over-adopt it. They don't. SE Ranking also tested the thing everyone actually wants to know, whether the file correlates with getting cited, using both statistical tests and an XGBoost model. The result was nil: removing llms.txt as a variable improved the model's accuracy, meaning the file was adding noise rather than signal.
You may see a higher number quoted: Ahrefs measured 28% adoption. That isn't a contradiction, it's a sample. Ahrefs surveyed its own web-analytics customers: a more technical, more AI-aware population than the web at large. The broader crawl is the better estimate of what everyone else is doing.
None of that makes llms.txt fake or pointless. It makes it a developer-tooling convention that hasn't crossed over into consumer AI search: a different audience, reading through a different pipe, for a different reason.
| llms.txt | Schema / structured data | Citable content | |
|---|---|---|---|
| Read by | AI coding tools, some dev-facing LLM products | Search + AI crawlers, broadly | Every crawler that indexes the page |
| Measured AI-search crawler traffic | 1.1% of requests; 97% of files never fetched | Fetched with every page crawl | Fetched with every page crawl |
| Confirmed use by ChatGPT/Gemini/Google answers | No public commitment | Established, foundational | Established, foundational |
| Effort to ship | Very low: one markdown file | Medium, JSON-LD per page type | High, ongoing writing discipline |
| What it fixes | Context-window efficiency for dev tools | Entity and fact legibility | What the engine actually quotes |
So should you still ship one?
Yes: just not for the reason most posts give you. Add llms.txt because:
- It's nearly free. A well-written one takes an hour, not a sprint.
- Your actual audience for it is narrower than "AI search." Coding agents: Cursor, Claude Code, GitHub Copilot, Windsurf, Cline, and Aider, look for
/llms.txtand/llms-full.txtby convention when a developer asks them to work against your docs. That audience shows up in the data: AI agents and agentic infrastructure drove 10.5% of llms.txt requests, roughly ten times what AI search retrieval bots did. If prospects or their engineers evaluate your product through those tools, llms.txt is read by the exact system doing the evaluating: today, not hypothetically. - It's a legible signal of technical maturity. A clean llms.txt next to a clean
/docstells a technical buyer something about how the rest of the org operates, independent of whether any crawler ever fetches it.
One caveat on tooling-driven adoption: Chrome added an llms.txt audit to Lighthouse in May 2026, which is a large part of why the debate reignited, and why some teams now see a checklist item where there is no measured benefit. In Ahrefs' data that audit accounted for about 1 in 1,000 fetches. A linter noticing the file is absent is not evidence that anything reads it.
Don't add it because you think it will move your ChatGPT or Gemini answer inclusion this quarter: the evidence doesn't support that yet, and treating it as a substitute for the fundamentals is the actual risk. If your GEO budget forces a choice, schema and entity work and content AI engines can quote are the levers with a confirmed line to what ChatGPT, Gemini, and Perplexity actually surface. For B2B software specifically, the evidence network that drives AI recommendations: documentation, review profiles, and comparison content, is where that budget belongs. llms.txt is the cheap, honest extra, not the strategy.
Frequently Asked Questions
Does ChatGPT or Google actually read llms.txt?
Not in any meaningful volume. Ahrefs analyzed 137,210 domains and found 97% of llms.txt files received zero requests in May 2026: no bots, no humans. Of the requests that did arrive, AI retrieval bots like OAI-SearchBot and PerplexityBot accounted for just 1.1%. No major AI provider has committed to reading the file, and Google's documentation states you don't need machine-readable AI files to appear in its AI features.
Does llms.txt improve AI citations or rankings?
No study has found an effect. SE Ranking analyzed roughly 300,000 domains and found no correlation between publishing llms.txt and how often a domain was cited by AI models, removing the variable actually improved their prediction model. Treat it as a developer-tooling convention, not a visibility lever.
How many sites publish an llms.txt file?
About one in ten. SE Ranking found 10.13% adoption across roughly 300,000 domains, with the largest sites adopting it least (8.27%). Ahrefs measured 28% among its own analytics customers, a more technical and AI-aware population than the web at large.
Is llms.txt an official web standard?
No. It's a community-proposed convention created by Jeremy Howard at Answer.AI in September 2024, published at llmstxt.org. It has not been ratified by the IETF, W3C, or any standards body: it's closer in status to an early sitemap.xml than to robots.txt.
Should I add llms.txt to my site?
It's worth the hour it takes if you have developer-facing documentation, since AI coding agents (Cursor, Claude Code, GitHub Copilot, Windsurf, Cline, Aider) do read it by convention. Don't expect it to move your visibility in ChatGPT, Gemini, or Perplexity answers directly: that's driven by schema, entity clarity, and citable content, not this file.
What's the difference between llms.txt, robots.txt, and sitemap.xml?
robots.txt tells crawlers what not to fetch. sitemap.xml lists every URL with no sense of priority. llms.txt is neither: it's a curated, human-written summary of what matters most on your site, formatted for a language model's limited context window rather than for indexing.
What should I put in my llms.txt file?
An H1 with your project or company name, a one-sentence blockquote summary, and H2-organized linked lists pointing to your most important pages (documentation, API reference, key product pages) each with a short description. Keep it curated and short; it's meant to be a reading list, not a second sitemap.
How Etched handles llms.txt
We ship it: it's cheap, it's harmless, and it serves the audience that does read it. But it's a line item in a GEO program, not the headline. The work that has a confirmed path to AI-answer inclusion is entity clarity, schema, and content built to be quoted, measured against your AI share of voice, not against whether a file exists.
Want to see where your brand actually stands? Run the free AI visibility check for an immediate automated snapshot, or review the human-led SEO/AEO Core Audit for a focused competitive baseline.
Run the free AI visibility check
Related Services
Last updated: October 21, 2026
