Corrections

When a scan gets something wrong — a false alarm, a misread file, a bad gate — and a site owner or reader flags it, we fix the scan and record it here rather than quietly editing the report.

Corrections log

Errors found after publication, with what caused them. Published rather than patched silently —

the corpus exists so that being wrong is recoverable and visible.

2026-10-10 — the published engine-vs-judgment gap is right on average and wrong where it is read

The leaderboard carries this caveat: "Our mechanical scoring engine scores the 55 sites rated both

ways about 7.5 points lower on average, and Edition 2 will publish reproducible engine scores."

The average is accurate. Re-measured across every Edition 1 site the engine has also scored:

The −7.6 at v1.0 is the published 7.5. Nothing about that sentence is wrong.

What it hides is that the gap is 2.4× larger exactly where readers look. The hand scores were

most generous at the top, so the named exemplars are the worst-calibrated entries on the site:

Five of the ten files presented as "what the top files do" would be D or F under the engine today.

A reader who takes "7.5 points" at face value expects 86 → 78, not 85 → 58 with a B becoming an F.

This is not engine immaturity. The leaderboard gap sits between −16.9 and −19.4 across four

engine versions and two rubric major versions. Decomposed at v2.1: judgment-to-engine accounts for

−5.5 of the corpus-wide gap and rubric tightening from v1.1 to v2.1 for only −1.3. The earlier

entry below (2026-07-20, judgment audits only half-close the gap) reached the same conclusion from

a smaller sample and named the cause: the hand scores were a holistic impression, and an impression

is most generous about the files it liked most.

The decision was already made and has not shipped. That same entry recorded: "retire the hand

scores rather than reconcile them … Edition 2 publishes engine scores and drops the hand numbers

entirely." That was 2026-07-20. Eighty-two days later the site still publishes the hand scores, now

beneath a homepage that says our numbers are re-derivable from stored bytes. The caveat is doing

work the correction should have done.

Two counts in the caveat are also off by one and worth fixing when Edition 2 lands: Edition 1 judged

59 sites, of which 54 have engine scores — not 55. The five never engine-scored are

perplexity.ai, expo.dev, docs.anthropic.com, langchain.com and www-dev.khov.com (a staging

capture, excluded by policy).

Unchanged: adoption rates, file sizes and every count on that page are parser-measured from the

archived files and depend on neither method. Nothing in this entry moves them.

Open: the spec for Edition 2 is at docs/superpowers/specs/2026-10-10-leaderboard-edition-2-design.md.

No published score changes until it ships.

2026-09-29 — third-party credentials removed from archived files

The archive keeps each site's files byte for byte, which meant it also kept credentials sites had

embedded in them. We now strip them. No score changed (tools/score.mjs --all output is

byte-identical before and after).

• sentry.io, 9 files captured 2026-07-19 (AGENTS.md, CLAUDE.md, the llms-.txt variants,

ai-catalog.json, ai-plugin.json, openapi.json): sentry.io answered those paths with its HTML app

shell, which embeds a Stripe publishable key and a Google Maps browser key. Both are public by

design, but we don't redistribute them: each value is replaced with [redacted-by-llmsrank].

• pages.json evidence (2026-09-29), 5 files (dev.hume.ai, developers.deepgram.com, you.com,

weather.com, wegmans.com): <head> links carried AWS presigned URLs, a browser apiKey and a

Google Maps key. tools/scrub-signed-urls.mjs removed the signing and credential parameters, and

every capture now does the same (tools/sanitize-url.mjs).

• Left as published: documentation placeholders inside sites' own llms-full.txt and OpenAPI files

(sk_live_xxxx, AWS's AKIAIOSFODNN7EXAMPLE, xoxb-your-token, example certificate keys).

2026-07-20 — llmsrankings.md is now generated output (task D3), and full-coverage sampling corrected the .md-twin rate

llmsrankings.md is rendered by tools/render-rankings.mjs from the store plus

corpus/observations/edition1-judgment.json — a labelled overlay holding the Edition 1

judgment scores (they existed only in the hand-written page; the overlay makes the page a pure

function of the store). Regeneration surfaced four corrections:

• `.md`-twin adoption 24% → 9%. The published figure sampled 37 sites (26/110). The D6 run's

evidence was never persisted; re-running across all 149 sites with an archived body

(tools/sample-md-twins.mjs, evidence in corpus/derived/md-twins-2026-07-20.ndjson) gives

40/445 = 9%. The original sample skewed toward doc-platform sites, which are exactly the

full adopters.

• Largest file is `twilio.com` at 2.17 MB (from stored bytes). The hand-written page said

2.28 MB in one place and called tollbrothers.com (1.36 MB) "the largest" in another —

internally inconsistent; the parser settles it.

• Blocked-cohort range is 18–29%, not 19–25%: the earlier figure predated the observations

backfill (D1).

• Board B had no rank #4 (a hand-numbering slip) and tollbrothers.com still showed its

superseded capped score in the homebuilder table after the 2026-07-20 cap-lift correction.

Both are now impossible by construction.

Also repaired: corpus/files/uhs.com/2026-07-19.meta.json contained invalid JSON escapes

(\. and \| in first_line) and could not be parsed; escapes removed, content unchanged.

2026-07-20 — three scores wrong, from trusting summaries over bytes

Task D5 rebuilt the file validator to skip leading HTML comment nodes before testing for an H1

(compass.com opens with <!-- version: 1.0.0 --> and had been wrongly quarantined). Re-running it

across the archived bodies contradicted three published scores. All three trace to the same root

cause: Edition 1 classified H1 presence from subagent summaries rather than from the stored

bytes. Two summaries said "no H1" about files that have one; one said nothing about a file that

doesn't.

Net effect on Board A: cvs.com drops out of the top three; indeed.com and tollbrothers.com

rise off the floor but stay in F. The headline findings are unaffected — adoption rates,

the compliance gap (1 file in 152), and the self-blocking sites are all unchanged.

Process change: H1 presence, link counts, description ratios and section counts are now derived

by parser from stored bytes, never from a summary. Any characteristic a summary asserts and the

bytes cannot confirm is UNVERIFIED (scoring.md §3.1).

2026-07-20 — capture and count corrections

• Four of five UA-refused domains recovered (task D4) with a rotated user agent: gm.com

(6 bytes, contents # test — the stub finding confirmed from bytes), t-mobile.com,

algolia.com, trivago.com. Only group1auto.com still refuses. Bodies now archived.

• Three more files have no H1, found on validation of the recovered bodies: t-mobile.com

(opens with --- YAML frontmatter), algolia.com (opens Algolia, unmarked), trivago.com

(opens with an ASCII-art logo). Total no-H1 files: 9 of 158.

• Present count 155 → 158, and AI-cohort adoption 62% → 64%, after the recovered captures

and the observations backfill (task D1). Top-100-US (13%), Fortune 500 (11.3%) and global-brand

(15%) rates are unchanged.

• Domains checked 727 → 723 after deduplicating domains that appeared in more than one cohort

(e.g. spotify.com, tripadvisor.com). The store deduplicates; the earlier figure double-counted.

2026-07-19 — capture-time corrections (recorded at the time)

• Archival UA was refused by five domains a browser UA reached; metadata retained, bodies not.

Resolved 2026-07-20 by D4 (above).

• Automated HTML detection quarantined compass.com, a valid llms.txt whose first two lines are

HTML metadata comments. Restored manually; validator fixed in D5.

2026-07-20 (later) — the no-H1 list was wrong, and the parser caught it

Task P0.2 built a real parser (tools/parse-llms.mjs) and ran it over all 156 archived bodies. It

disagreed with the no-H1 list published hours earlier in the same day's corrections.

Root cause: my shell check tested whether the first meaningful line was an H1. The spec

requires an H1 to be present, not to be first. Files that open with a blockquote, a YAML block

or an ASCII-art banner and carry their H1 a few lines down were wrongly flagged.

Corrected list: 5 of 156 — algolia.com, cigna.com, cvs.com, notion.com, twilio.com.

Process change: file characteristics are derived by tools/parse-llms.mjs from stored bytes.

Ad-hoc shell greps are for exploration only and must never reach a published number. The parser is

deterministic (verified: two runs produce byte-identical output), so any figure it produces is

reproducible by anyone with the corpus.

Parser vs. published claims — everything else held

2026-07-20 — the mechanical engine disagrees with the published rankings

Task P0.4 implemented tools/registry.json + tools/score.mjs — the rubric in scoring.md

executing against parsed bytes and archived evidence. Compared against the 20 hand-assigned scores

published in Edition 1:

• Engine is lower on 19 of 20, mean delta −12.9 points.
• Rank agreement (Kendall tau) is 0.14 — close to random. The two methods do not merely differ

in level, they disagree about order.

Largest divergences: vercel.com 82→48.8, posthog.com 85→55.6, pinecone.io 83→59.8,

docs.stripe.com 80→57.4. Smallest: zapier.com 86→81.8, khov-dev 77→72.8.

What this means, stated plainly: the Edition 1 leaderboard is judgment, not measurement. The

hand scores were assigned by reading files and forming a view. That is defensible as expert

assessment and indefensible as a reproducible metric — which is exactly the standard this project

has been holding other trackers to.

But the engine is not yet the answer either. Its weakest audits are the ones that need language

understanding and are currently regex proxies: description quality (C2), API documentation depth

(A5), agent-instruction substance (A1/A3), FAQ detection (A7). A file can document 70 endpoints

with worked examples or mention the word "endpoint" once, and the current A5 cannot tell them apart —

which is why docs-heavy sites are penalized hardest.

Resolution (next iteration): implement the LLM-assisted audits that tasks.md already assigns —

Haiku 4.5 for description quality at crawl scale (P1.7), Sonnet 5 for agent-instruction and API-depth

assessment. Then re-run and re-measure agreement. Until that lands:

• Edition 1 rankings stay published but are labelled judgment-derived, not reproducible.
• Engine scores live in corpus/derived/scores-2026-07-19.ndjson and are not published as a

replacement, because a low tau means we cannot yet say which is closer to truth.

• Nothing about the load-bearing findings changes — adoption rates, the compliance gap (1 file in

156), self-blocking sites and the no-H1 list are all mechanical and unaffected.

Tuning the engine's constants until it reproduces my hand scores was available and rejected: that is

fitting the ruler to the answer.

2026-07-20 — judgment audits only half-close the gap, which is itself the finding

Task P1.7/A5 built tools/audit-llm.mjs (Haiku 4.5 for description quality at crawl scale, Sonnet 5

for API depth and agent-instruction substance, both prompt-cached and schema-constrained) and wired

an audit overlay into the scoring engine. No API credentials are available in this environment, so

the hypothesis was tested with a hand-authored calibration overlay across 20 sites, clearly marked

source: manual-calibration.

Judgment audits move things in the right direction — Pinecone +12.2, PostHog +7.3, UBS +7.0 — and

they are clearly the correct implementation. But they explain less than a quarter of the gap.

Rank agreement stays poor.

The conclusion I did not want: the divergence is not mainly an implementation shortfall. The

rubric, applied mechanically with good inputs, disagrees with my hand scores about which files are

better. Two things can be true and I believe both are:

1. The rubric's weights and scoring functions do not capture what I intuitively valued when reading

these files.

2. My hand scores were a holistic impression rather than a disciplined application of the rubric —

which is exactly what an expert assessment is, and exactly what a metric must not be.

Decision: retire the hand scores rather than reconcile them. The engine output is reproducible,

evidence-backed, applicability-gated and inspectable; the hand scores are none of those things.

Presenting both invites the reader to pick their favourite. Edition 2 publishes engine scores at

rubric_version 1.0 and drops the hand numbers entirely.

What this does not resolve: whether rubric v1.0 weights are right. That is not answerable by

more of my opinion — it needs the differential analysis in analysis.md §2 (top-decile vs

bottom-half against real outcomes) and ultimately the phase-3 log signal. Until then the score is

honestly labelled: a reproducible measurement against a stated rubric, not a claim about which files

get cited more.

Continuing to tune constants toward my priors was available at every step and rejected each time.

2026-10-10 — the published figures were July's, and three things kept them that way

Everything on the site describing "our research corpus" was measured in July. It is October. The

numbers are now re-measured, and the act of re-measuring found more than it changed.

Three defects stood between a fresh capture and a published number. Each would have quietly

republished July while appearing to work. parse-llms.mjs chose a domain's llms.txt by readdir

order, which is alphabetical, so 2026-07-19 beat 2026-10-10 and the October files were never

read; its parsed_at was the literal string 2026-07-20. audit-stats.mjs scored whatever

score.mjs defaulted to — the July parse — while stamping the file with the newest capture date it

found on disk, so the label and the numbers came from two different places and had already drifted

apart. capture-llms.mjs could only walk corpus/files, so a sweep could not reach the 565 cohort

domains that have no directory, and using it to refresh adoption would have shrunk the denominator

to the domains that already had a file.

A 200 with an empty body was recorded as a published file. sherwin-williams.com answered HTTP

200 with zero bytes. Stored as PRESENT it counted as an adopter and scored 0, pulling the corpus

mean down with a file nobody wrote. An empty body is now a determined absence — the server

answered, there is no file — with the real status kept in the meta. This is the same mistake as

recording a 403 as ABSENT_404, in the other direction: the first invented non-adopters, this one

invented an adopter.

Adoption was understated, and still is on the headline basis. A sweep returns five outcomes and

only three answer "does this domain publish an llms.txt". BLOCKED and UNVERIFIED mean we were

refused or could not reach it. July's rate divides present by every domain checked, which records

all 203 of those as non-adopters. The observations file now publishes both: rate on July's basis

so the runs stay comparable, and rate_determined over the domains we actually determined. Among

the top 100 US websites that is 18% against 23%; among the Fortune 500, 15% against 21%.

What actually changed in the world, stated carefully. 182 files against July's 158. 39 are new.

14 domains that had a file in July no longer show one, but only three of those are determined

absences (expo.dev, langchain.com, kia.com); ten went bot-blocked and one unreachable. We did not

learn that those eleven files are gone, only that we can no longer read them, and reporting them as

removals would have invented a decline.

Scores moved up and the corpus average moved down, both correctly. Paired on the same 147

domains and the same rubric v2.1, the mean goes 58.7 → 60.4, +1.7 points. The whole-corpus mean

goes 58.1 → 59.2, because the 40 files new to the corpus average 54.7 — well below the 60.4 of the

files we can track. Sites that already had a file improved; the newcomers start lower and pull the

average down. Either number alone misreads it, so both are published.

The loudest signal is not adoption at all. 31 domains went BLOCKED → ABSENT_404 and 24 went the

other way, with 20 more ABSENT_404 → UNVERIFIED. Edge bot policy churns harder than publishing

behaviour does, and it moves our denominator. Any single-run adoption figure, ours included, is

partly a measurement of who was willing to answer a bot that day.

The maintenance problem this exposed. Roughly 35 per-audit figures were hardcoded in prose

across audit-faqs/, audit-docs.ts and the learn pages — "the check applies to 83 of 156 files;

the mean is 0.233" — all of which go stale on every scan and were updated here by hand. /learn/research

computes its figures from audit-stats.json and could not drift. The prose should be generated the

same way; until it is, every scan carries this risk again.