llms.txt: what it is and whether it really helps SEO

Google put it in its own documentation: it does not use your llms.txt. And yet the most debated file of 2026 does have a real consumer, just not the one you were sold.

The useful question is not what it is but whether it works. Here is the current spec, Google's exact wording, the largest log study of the year and a decision rule you can apply today.

What llms.txt is (and what it is not)

llms.txt is a proposal by Jeremy Howard (Answer.AI), published on September 3, 2024 and maintained at llmstxt.org, with a v2 revision dated August 10, 2026. The idea is simple: a Markdown file at the site root, or at any subpath covering what sits below it, that hands a language model a curated map of the site. A short summary plus links to the pages that matter.

What it is not, and this is where half the hype falls apart:

It is not a permissions file. It grants and forbids nothing.
It is not a ranking signal.
It is not a sitemap, even if it looks like one.
It does not guarantee that any engine reads it.

The format the v2 spec actually asks for

Order matters if you want a valid file:

An optional BOM.
One H1 with the project name. It is the only required section.
A blockquote holding the summary.
Free-form Markdown sections with context, no headings.
Zero or more ## sections with link lists. Every item is a required Markdown link, with optional notes after a colon.

The most useful convention is ## Optional, which marks material an agent can skip when it needs less context. A minimal example:

# My project

> API documentation and guides.

## Docs
- [Quickstart](https://example.com/quickstart): setup in five minutes
- [API reference](https://example.com/api): endpoints and errors

## Optional
- [Blog](https://example.com/blog): release notes

A file with no links is a technically valid, practically useless stub. The v2 also widens the scope: it adds per-page .md mirrors and the relationships rel="alternate" type="text/markdown" and rel="describedby", which a build can generate without manual work.

llms.txt is not robots.txt

robots.txt states what you may crawl, and search engines have obeyed it for thirty years. llms.txt states where to start reading, and nobody is obliged to look. One is a rule with consequences; the other is a courtesy note. The first one is covered in what the robots.txt file is and in how to create a correct robots.txt in 2026.

One confusing detail: llms-full.txt is not in the spec. It is a tool convention that concatenates all content into a single file.

What Google says, word for word

Google's "Optimizing your website for generative AI features on Google Search", updated July 10, 2026, includes a myth-busting section. Direct quote: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It goes further: creating one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Its AI features documentation repeats it: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

One nuance blocks the wrong conclusion: Googlebot did request the file about 900 times in May 2026, but only because it crawls any URL it discovers. Requesting a file is not using it. And Google-Extended is not a crawler: it is a token you write into robots.txt.

If Google ignores it, the value has to sit with other consumers. Those can be measured in logs too.

The data: 137,210 domains and 97% that nobody asked for

Ahrefs published the largest log study on this file on June 15, 2026: 137,210 domains, using May 2026 records.

28% of domains publish a llms.txt, and that is a ceiling: the sample skews technical.
Of the roughly 38,000 with a valid file, 97% received not a single request in all of May.
The remaining 3%, about 1,100 domains, absorbed all the traffic: some 22,000 requests, 96% of them bots. The breakdown deflates the myth: SEO audit tools 21.7%, other or unidentified 14.9%, general crawlers 13.1%, technical profiling 11.6%. AI bots together add up to 19.5%, with training at 5.3% and search retrieval at just 1.1%.
OAI-SearchBot, PerplexityBot and Claude's search crawler combined for a couple of hundred requests spread across thousands of sites.

Methodological warning: requesting the file is not reading it, so those figures are a ceiling and not a floor.

The second study comes from SE Ranking, across roughly 300,000 domains: 10.13% adoption and no correlation between having the file and being cited by models. Removing the variable from their model actually improved accuracy. Noise, not signal. Their own wording: "doesn't seem to directly impact AI citation frequency. At least not yet."

Who does read it: coding agents, not answer engines

The most useful finding is a different one. Claude Code out-requested every retrieval, assistant and training crawler combined. Agents and their infrastructure accounted for 10.5% of requests, more than any other AI bot category.

Translated into a decision: the file helps if your audience is developers or your site is technical documentation. It does not help you "show up in ChatGPT".

No bot goes looking for a file that is not there

Requests to /llms.txt that returned a 404 were 98% human traffic and zero AI bots. The ones knocking were people checking out the competition.

The consequence is uncomfortable for anyone selling the idea: publishing the file puts you on nobody's radar. Systems ask for it when a link, an index or the user tells them it exists.

Should you ship it? The decision rule

Ship it if:

Your build generates it on every deploy, with no manual work.
Your audience is devs or agents: docs, APIs, libraries.
You want clean .md mirrors of your key pages.

Skip it if:

You expect clicks or citations from Google. The data says they do not come from here.
You would maintain it by hand, because once it costs human time the return turns negative.
Your time is better spent on indexing, content and speed.

Technical cost: zero. Risk in Google: zero, because it ignores the file.

Lighthouse audits it, and that is not a ranking signal

Lighthouse 13.3, released May 7, 2026, added an Agentic Browsing category to its default config, including an llms.txt audit. It reached Chrome 150 DevTools and PageSpeed Insights about two weeks later.

It is a presence and format check, not a ranking signal. If the file does not exist the audit does not apply, so it costs you no points. Green means the file is there and parses, not that any engine reads it.

Where the real debate is: robots.txt

The file every crawler already requests and obeys is robots.txt, and that is where control over AI content use is actually moving:

Cloudflare Content Signals, from September 2025: a Content-Signal field inside robots.txt with three permissions, search, ai-input and ai-train.
IETF AIPREF: a vocabulary of AI usage preferences, also built on robots.txt and still in draft.

The honest reading: the industry is consolidating access control in the file that has existed since 1994, not in a new one. That leaves llms.txt as a courtesy map, not a permissions switch. Getting your site ready for browsing agents is covered in how to make your website agent-ready.

A five-step checklist

Write the goal down: coding agents or AI visibility? If it is the second, this file is not the lever.
Generate it in the build, never by hand: a script that walks your key pages and writes the H1, blockquote and sections.
Go for the full format, not a stub: H1, summary and at least one section with links.
Measure before you believe: filter /llms.txt requests in your logs for 30 days and compare against baseline.
Do not neglect what actually moves the needle: indexing, structured data for rich results, Core Web Vitals and Search Console. The detail is in what rich snippets and schema markup are.

Bottom line

Google ignores it and says so in writing. Almost nobody requests it: 97% of valid files in the largest log study got zero requests in a month. And its only demonstrated consumer is coding agents, not the answer engines you were promised.

Ship it if it comes free in your build and serves your own agents. Do not sell it, and do not buy it, as an AI visibility strategy: it is cheap hygiene, not a visibility lever. If appearing and getting cited by generative engines is the goal, the real work is in GEO: how to appear and get cited in ChatGPT, Perplexity and Gemini.