# The llms.txt year in review, 2026

Three things happened in 2026: a major browser vendor started auditing `llms.txt`, the biggest CDN made blocking AI crawlers a default, and the independent tool vendors in this category were absorbed into enterprise suites. Meanwhile 187 published files scored a mean of 59.2 on our rubric, with 91% landing at D or F. This is what we measured, what we did not, and what we got wrong.

Updated 2026-10-06. This is the markdown version of https://llmsrank.dev/learn/llms-txt-2026.

## What changed in 2026

The argument through 2025 was whether anyone consumed `llms.txt` at all. That question closed in a more interesting way than either side expected.

1. **May: Google began auditing it.** Chrome added an `llms.txt` check to Lighthouse under a new category it calls agentic browsing, alongside WebMCP and a subset of accessibility audits — with documentation on how to write the file.
2. **The same guidance says it changes nothing for search.** Google published that `llms.txt` has no effect, positive or negative, on Search rankings or AI Overviews. Both halves are true: it is an agent-readiness signal, not a ranking one, and anyone quoting only one half is misleading you.
3. **July 2025 onward: blocking became a default.** The largest CDN made blocking AI training crawlers the default for new domains, then in September 2026 replaced its single toggle with separate Search, Agent and Training controls. New sites and untouched free-tier accounts can be refusing AI crawlers without anyone having chosen it.
4. **The independents were bought.** Within one quarter, two of the three vendors we had catalogued as independent startups became parts of enterprise suites, and a legacy SEO incumbent’s AI product became the intelligence layer inside one of them.

> **Note: The deliberately unscored part.** Google’s agentic browsing category produces no 0–100 score. Its documentation says the standards are still emerging and the aim is to gather data and give actionable signals rather than a ranking. That is a reasonable position, and worth holding next to any number — ours included.

## What 187 published files actually look like

We archived the raw bytes of every file and rescored all of them with the current engine (rubric v2.1, files captured through 2026-10-10). Mean score **59.2**, median **59.9**.

| Grade | Sites |
|---|---|
| A | 0 |
| B | 1 |
| C | 16 |
| D | 76 |
| F | 94 |

**91% score D or F.** That sounds like a damning result and it is not quite one: the pattern underneath is consistent and fixable.

> **Note: The finding of the year, in one line.** Everyone can serve the file. Almost nobody fills it in.

The plumbing is solved. Every file we could fetch was served with a usable content type, fetched cleanly, and covered the URL it should. Encoding and parsing is at 99%, and the two structural basics — a single H1 and H2 sections — clear 88% and 90%.

What is missing is everything that speaks to an agent rather than a parser. **Not one file in the corpus earned full marks on entity identity** — saying plainly who you are — and 78% scored zero on it. Intent routing, data-freshness caveats, link relations and documented API error handling are all at or near zero full marks. Four in five files publish no agent instruction file at all.

The per-audit figures, including everything summarised here, are published and regenerated from the same data on [the llms.txt Index](https://llmsrank.dev/learn/research), with the raw aggregate at [`/research/stats.json`](https://llmsrank.dev/research/stats.json).

## Who publishes one

Adoption is still concentrated where the readers are. Among the top 100 AI companies, 64% publish a file. Among the top 100 global brands it is 15%, among the top 100 US websites 13%, and across the next 400 of the Fortune 500 it is 11%.

We also pointed our own link and markdown checks at seven companies selling AI-visibility tools, ourselves included. Three publish a well-formed file, three publish none, and one publishes a file in which four of its 103 links still resolve. We are not naming them: each is a half-hour fix and some are likely fixed already. The point is narrower and more useful — **a file nobody validates is a file that quietly stops working**, and the broken one was generated with the wrong URL prefix and apparently never checked.

## Four things that move AI visibility which we cannot measure

The most useful thing a measurement product can publish is the edge of what it measures. A perfect score on our rubric says a site is retrievable, parseable and machine-legible. These are mechanisms it does not touch.

- **Did a crawler actually come?** Fetch rate from server logs is the leading indicator three independent vendors now name. We check whether a crawler *would* be served; we cannot see whether one arrived.
- **Citations decay.** Being cited today is not being cited next month, and the decay appears to be engine-specific. Any two-point comparison confounds decay with whatever you changed.
- **One prompt becomes many queries.** Engines expand a question into several retrieval queries before answering, so the prompt you tested is not the query that decided the answer.
- **Being cited is not being used.** Research published this year separates whether a page is cited from whether it shaped the answer — and finds one engine citing 2.4× more sources at roughly a quarter the influence each.

There is also a large category we deliberately do not touch: off-site mention. Being ranked first in someone else’s comparison article measurably moves AI visibility, and nothing on your own site changes. We measure the site. That is a choice, and it should be stated rather than left for a customer at full marks to discover.

## What the independent research says

This field acquired a critical literature in 2026, and its conclusions are more sobering than the marketing around it.

> No reviewed technique shows a stable, longitudinal, cross-platform causal effect on organic discoverability or downstream behavior.
>
> Source: [Martinez — A Critical Survey of Generative Engine Optimization (2023–2026)](https://arxiv.org/abs/2607.14035)

That survey covers 45 studies. It also finds the widely-quoted "40% improvement" from the field’s founding 2024 paper to be conditional on a source already being present in context — establishing neither discoverability nor durable traffic. We stopped citing that number.

The strongest positive result points at the unglamorous half. A fitted model over 34,960 observations found brand mention rates rising from under 4% with no domain presence to above 90% when both a site’s own pages and third-party mentions are retrievable — with owned-domain exposure alone worth 40 to 49 percentage points. Being fetchable and legible is the part with evidence behind it.

## What we got wrong this year

We ran a monthly comparison on one site and watched a 20.8-point citation gain dissolve under its own negative control, ending at −6.3 points on the pre-registered treated-against-untreated comparison. We published the null result rather than the headline.

We also shipped a markup-depth metric that counted script and icon tags as nesting levels, reported our own pages at 48 levels when they are 13, and believed it — because it confirmed something we were already prepared to believe about ourselves. A review caught it. [The longer version of both stories](https://llmsrank.dev/learn/what-we-got-wrong) is published, as is [every correction and its cause](https://llmsrank.dev/docs/corrections).

This field is about eighteen months old. The useful posture is not confidence; it is being checkable. Every figure on this page is derived from stored bytes by a [versioned rubric](https://llmsrank.dev/docs/method) and carries the date it was measured.

## Frequently asked questions

### Is llms.txt worth publishing, given Google says it does not affect rankings?

Those are two different questions. Google states it has no effect on Search rankings or AI Overviews, and in the same year began auditing it under agent readiness. If your aim is ranking, it is not the lever; if your aim is being usable by agents and assistants, it is the file they look for first.

### Why do so many files score D or F?

Because the hard part is not the file, it is the content. Across 187 files the plumbing is near-universal — correct content type, clean fetch, parseable markdown — while the sections that speak to an agent are mostly absent. Not one file earned full marks for stating plainly who the organisation is.

### Could my site be blocking AI crawlers without my knowing?

Yes, and this is the most commonly missed failure we see. CDN and firewall settings can refuse AI crawlers at a layer your robots.txt never mentions, and for some providers blocking is now a default for new domains. A scan checks whether crawlers are actually served, not just whether robots.txt permits them.

### How is this report different from the llms.txt Index?

The [Index](https://llmsrank.dev/learn/research) is the live per-audit reference, regenerated from the corpus. This report is the year: what changed outside our data, what the corpus shows, and what we could not measure. Both are computed from the same figures.

### Can I check these numbers myself?

Yes. The aggregate is published at `/research/stats.json`, the scoring rules and their version are on [the method page](https://llmsrank.dev/docs/method), and every correction we have made is listed with its cause. The vendor check in this report was run on 2026-10-05 with a browser user agent, following redirects.

## Sources

- [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)
- [LLMSRank — how scoring works](https://llmsrank.dev/docs/method)
- [LLMSRank — corrections log](https://llmsrank.dev/docs/corrections)
- [Chrome for Developers — Lighthouse agentic browsing: llms.txt](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)
- [Cloudflare — new options to manage AI traffic](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)
- [Martinez — A Critical Survey of Generative Engine Optimization (2023–2026)](https://arxiv.org/abs/2607.14035)
- [Zhang et al. — From Citation Selection to Citation Absorption](https://arxiv.org/abs/2604.25707)
- [Tannenbaum — A Fitted Stage Model of Brand Visibility in AI Search](https://arxiv.org/abs/2609.23162)

## Related

- [The llms.txt Index: how 187 sites score](https://llmsrank.dev/learn/research.md)
- [What we got wrong](https://llmsrank.dev/learn/what-we-got-wrong.md)
- [Does ChatGPT, Claude, Perplexity or Google read llms.txt?](https://llmsrank.dev/learn/does-ai-read-llms-txt.md)