What we got wrong
We grade other people's sites, so our own numbers have to survive being checked. These are cases where ours did not: a null result we had hoped would be positive, a metric that was measuring the wrong thing, and claims in this field — including ones we repeated — that aged badly within months. Each is here because the alternative is a silent patch.
A null result we published instead of burying
We ran a monthly comparison on one site to see whether publishing an llms.txt moved how often AI assistants cited it. The headline looked good: citation rate up 16 points on one engine, and up 20.8 points in the tier the change targeted.
It did not survive its own controls. The negative control — a set of prompts the change should not have touched — moved too. The gain was confined to a single engine while another went slightly down. And the pre-registered comparison of treated against untreated pages came out at −6.3 points: the untreated ones improved *more*.
We wrote that up as a null result, with the method and the raw records, rather than reporting the 20.8. A measurement that only survives when you skip the control is not a measurement.
A metric that measured the wrong thing — including about us
Building a check for how deeply a page nests its markup, we shipped a counter that treated every unclosed tag as a nesting level. HTML lets <p>, <li> and <td> omit their closing tags, so a flat list of 100 items read as 100 levels deep. We caught that one and fixed it.
The fix was incomplete, and the second bug was worse: elements whose contents are skipped wholesale — <script>, <style>, <svg> — were never removed from the count. Every icon and every script tag on a page added a phantom level.
We did not notice, because the number was plausible. We ran the check against our own site, saw it fail, compared ourselves to well-built reference sites, and concluded our markup was unusually deep. We reported that internally as a finding about us.
| Page | What we reported | Actual |
|---|---|---|
| Our home page | 48 levels | 13 |
| Our method page | 37 levels | 14 |
| A large reference documentation site | 17 levels | 16 |
A review caught it. Our pages are not deep; they are *shallower* than the reference site we had measured ourselves against. The finding was withdrawn. The uncomfortable part is not the bug — it is that a wrong number survived because it confirmed something we were prepared to believe about ourselves.
Claims in this field age badly — including ones we repeated
In 2025 a widely-quoted position held that no AI service had said it used llms.txt, and that there was therefore nothing to gain from publishing one. That was accurate when written. In May 2026 Google added an llms.txt audit to Lighthouse, under a category it calls agentic browsing, with documentation on how to write the file.
The honest reading is narrower than a reversal. Google also published guidance that the file does not affect Search rankings or AI Overviews — so the original argument about *search visibility* stands, and what changed is that agent-readiness became a thing a major vendor measures. Both halves are true, and quoting either alone misleads.
The same applies to the number everyone in this field repeats. A 2024 academic paper is routinely cited for a "40% visibility improvement". Checking it, we found three secondary sources attributing that figure to three different techniques, and one reporting a *negative* effect for pages already ranking first. A 2026 survey of 45 studies then found the original gains were conditional on a source already being in context.
No reviewed technique shows a stable, longitudinal, cross-platform causal effect on organic discoverability or downstream behavior.
We stopped quoting the 40%. We would rather publish a smaller number we can re-derive than a larger one we cannot.
We ran our own checks on the people selling AI visibility
On 5 October 2026 we pointed three of our own audits — link hygiene, agent-friendly links and markdown versions — at the published llms.txt of seven companies selling AI-visibility tools, our own corner of the market included.
| Result | Vendors |
|---|---|
| Publishes a well-formed llms.txt with working links | 3 of 7 |
| Publishes one where 4 of 103 links resolve | 1 of 7 |
| Publishes none at all | 3 of 7 |
| Serves markdown versions of its pages | 1 of 7 |
We are not naming them. Every one of those findings is a half-hour fix, several may already be fixed by the time you read this, and the point is not who was caught — it is that a file nobody validates is a file that quietly stops working. The broken one was generated with the wrong URL prefix and, as far as we can tell, never checked.
Why any of this is on our website
Because the alternative is asking you to trust a score. Every number we publish is derived from bytes we stored, by a rubric with a version, and when one turns out to be wrong the fix and its cause go in a public corrections log rather than a silent edit.
- An audit that cannot be re-run on the same inputs is an opinion.
- A number without a date is a number about a web that has since changed.
- A result that only holds without its control is not a result.
This field is about eighteen months old. Nobody in it, us included, should be confident; the useful thing is to be checkable. That is why we keep the records — and why being wrong in public is part of the product rather than an embarrassment to manage.
Frequently asked questions
Why publish your own mistakes instead of quietly fixing them?
Does the null result mean llms.txt does not work?
Why will you not name the vendors whose files were broken?
If claims in this field age badly, why should I trust yours?
How often does this page change?
Sources
- llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)
- LLMSRank — corrections log
- LLMSRank — how scoring works
- Chrome for Developers — Lighthouse agentic browsing: llms.txt
- Aggarwal et al., GEO: Generative Engine Optimization (KDD 2024)
- Martinez, A Critical Survey of Generative Engine Optimization (2023–2026)