Last updated: July 2026
If you sell llms.txt tooling — and we do — the tempting move is to show you a chart of AI traffic going up and imply the file is behind it. We would rather show you the actual numbers, because they are less flattering and more useful. If you first want to know what the file is, start with What is llms.txt?; this article is only about whether it works.
Short version: most AI crawlers ignore the file today, one search giant has said so on the record, and yet there are specific, verifiable places where llms.txt is being read and used right now. Both halves of that sentence matter.
The uncomfortable numbers first
Two independent studies from 2026 measured what AI bots actually do, and neither is kind to the hype.
OtterlyAI's GEO experiment (February 2026) monitored server logs on a test site for 90 days. Out of more than 62,100 visits from AI-branded bots, exactly 84 requests touched /llms.txt — about 0.1% of AI bot traffic. The file received no higher crawl priority than static assets like PDFs, and its presence produced no measurable change in AI search visibility.
Ahrefs analyzed 137,210 domains (June 2026) and found that 97% of published llms.txt files received zero requests in the month studied. Among the files that did get traffic, SEO audit tools were the largest readers (21.7% of requests) — tools checking whether you have the file, not AI systems using it. All AI bots combined accounted for 19.5%. And notably: AI bots never requested llms.txt files that did not exist. They do not go looking for it; they only fetch it when something points them there.
Then there is Google. Gary Illyes confirmed at Search Central Live in July 2025 that Google Search does not support llms.txt and is not planning to; Google's 2026 guidance for AI features in Search lists it by name as a tactic you can skip. John Mueller has compared it to the keywords meta tag. For ranking in Google Search, AI Overviews included, the file does nothing.
Neither OpenAI nor Anthropic has committed to reading llms.txt in their production crawlers either. Their official crawler documentation (OpenAI, Anthropic) describes robots.txt handling in detail and says nothing about llms.txt. That silence has held through mid-2026. Anyone telling you the major AI platforms officially consume this file is ahead of the facts.
So if the question is "will adding llms.txt boost my AI search visibility this quarter?" — the honest answer is no, and any vendor claiming otherwise is selling against the data.
Where llms.txt demonstrably works today
The same studies that produced the ugly numbers also show where the file genuinely gets used. Three places stand out.
AI coding assistants and developer tooling
This is the strongest signal in the data. In the Ahrefs study, Claude-Code — the fetcher behind Anthropic's coding agent — out-fetched most dedicated AI retrieval bots. When a developer asks their coding assistant about a library, the assistant frequently pulls that project's llms.txt to find the right documentation pages fast.
The documentation ecosystem built around this behavior early. Mintlify enabled automatic llms.txt generation for every docs site it hosts back in November 2024 — including the documentation of Anthropic and Cursor themselves — and reports that nearly half of documentation traffic now comes from AI agents like Cursor, Claude Code, ChatGPT, and Perplexity. For developer-facing content, llms.txt is not speculative. It is in active use.
On-demand fetchers, not training crawlers
There is a distinction the headline numbers blur: the difference between training crawlers and user-triggered fetchers.
Training crawlers (GPTBot, ClaudeBot) sweep the web at scale to collect model training data. These are the bots the studies mostly measured, and they largely skip llms.txt. But both OpenAI and Anthropic also operate on-demand fetchers — ChatGPT-User and Claude-User — which visit your site the moment a user asks the assistant a question about it. When an assistant browses your site live, a curated markdown file at a predictable path is exactly the kind of resource it can use, and agentic fetchers made up the largest AI slice of llms.txt requests in the Ahrefs data. This is the reading pattern llms.txt was designed for: inference time, not training time.
Platforms are starting to bet on it
In May 2026, Shopify started serving /llms.txt automatically on every active storefront, alongside agents.md and its agentic-commerce endpoints — part of the same push that made Shopify catalogs browsable by AI agents in ChatGPT, Copilot, and Perplexity. A platform hosting millions of stores does not ship a default file for a convention it considers dead. (If you run a Shopify store, the auto-generated default is thin — our Shopify guide covers how to replace it.)
Even inside Google the picture is less uniform than the Search team's dismissal suggests: Lighthouse 13.3 shipped an experimental agentic-browsing audit in May 2026 that checks for llms.txt. Search ranking and agent-readiness are different problems, and Google's own tooling treats them differently.
The honest case for having one anyway
Given all that, why bother? Four reasons, none of which require believing the hype.
The cost is near zero. Generating the file takes minutes, placing it is a one-time job, and it needs a refresh only when your site changes meaningfully. Weigh a few minutes of setup against an uncertain but non-zero payoff, and the math is forgiving.
The file is useful independent of the standard. A good llms.txt is a curated "here is what we do and where to find it" snapshot in plain markdown. Any agent that lands on your site — an AI assistant answering a customer's question, a shopping agent, a tool you have never heard of — can use that, whether or not it formally "supports llms.txt." The Ahrefs data showing agentic fetchers as the biggest AI readers suggests this is already how the file gets consumed.
Placement is one-time; adoption is a variable. The bet is asymmetric. If adoption stays flat, you spent minutes. If it tips — and Shopify shipping it as platform infrastructure is the kind of move that tips conventions — you are already in place while others scramble.
It forces a useful exercise. Writing down what your business does and which ten pages matter most is valuable even if no bot ever reads it. Most sites have never done this.
To be equally honest about the other side: if you expect ranking gains in Google or a visibility jump in ChatGPT next month, skip it. That is not what the data supports.
Our receipts, coming
One more thing, since this article is about evidence: llms.fyi hosts the llms.txt files of our Automated Mode customers, which means we see exactly which AI bots fetch those files, how often, and when. We are collecting that data now and will publish aggregated numbers — including the unflattering ones — as the picture firms up. If the fetch rates stay low, we will say so. That is the deal.
Verdict: it depends on what your site is
Developer tools, APIs, libraries, documentation sites. Yes, clearly. Coding assistants fetch llms.txt today and your users are asking them about you. This is the proven use case.
Shopify stores. Yes — you already have one whether you like it or not. The only question is whether you keep the generic default or replace it with something that sells your store.
Ordinary business websites. Worth it, with honest expectations: not for search rankings, but as a cheap, curated entry point for the assistants and agents that increasingly answer questions about you. On WordPress, Webflow, and Squarespace, placement takes minutes.
Sites chasing a Google ranking boost. No. Google has said plainly it does not use the file. Spend the time on content instead.
If you already have a file, run it through our validator to check it against the spec — and while you are at it, check your robots.txt to make sure you are not accidentally blocking the very fetchers that would read it. The two files do different jobs; we break that down in llms.txt vs robots.txt.
Frequently Asked Questions
Do AI crawlers read llms.txt?
Mostly not, today. Training crawlers like GPTBot and ClaudeBot overwhelmingly skip it — that is what the Otterly and Ahrefs studies measured. The meaningful readers are AI coding assistants and on-demand fetchers acting on a user's request, which is the use the file was designed for.
Does llms.txt help SEO?
No, not for Google Search. Google has stated on the record that it does not use llms.txt for Search or AI Overviews. Whatever value the file has lives outside traditional search rankings.
Has anything changed in 2026?
Yes, in both directions. The negative studies (Otterly, Ahrefs) were published in 2026 — but so was Shopify's platform-wide rollout and Lighthouse's agentic-browsing audit. Adoption is moving; blanket "it's dead" and "it's essential" takes are both out of date.
Should I remove my llms.txt if the numbers are this bad?
No. It costs nothing to keep, some agents do read it, and removing it only guarantees a zero. Just make sure it is accurate — a stale file is the one way it can actively hurt you.