Skip to content
LLMSRank
Docs menu

Token budget

Token budget and context fit

In short: Keep the file under about 8,000 tokens (roughly 32,000 bytes): trim low-value links and move long reference text into linked pages.

  • Category: Curation Quality · weight 12 of 100 within the category
  • Curation Quality is worth 25 in generic and commerce, 20 in regulated and technical (points of 100, by scoring profile)
  • Audit ID C4 (as shown on reports) · rubric v2.1

What it checks

Whether the file fits an agent’s context budget, about 8,000 tokens.

Why it matters

An agent reads the whole file before choosing links. A file too large to fit gets truncated, and the truncated part may be what mattered.

When it applies

Always, for any llms.txt that exists.

An audit that doesn't apply, or that we couldn't verify, is removed from your score and its weight shared among the Curation Quality audits that do apply. It is never scored as a zero.

How it's scored

1.0 ≤ 8k tokens · scaled down past that (1 − (tokens − 8k) / 34k) · 0.3 > 25k. Tokens are estimated as the file’s UTF-8 bytes ÷ 4, so 8k tokens is about 32,000 bytes. Budget only: an ## Optional section doesn’t change the score.

How to fix

  1. Aim for under 8,000 tokens (roughly 6,000 words).
  2. Cut duplicated, low-value and archive links first (see C1).
  3. Move long reference material into linked pages instead of the file itself.

Frequently asked questions

How does LLMSRank count the tokens in my llms.txt?
It estimates: the file’s size in UTF-8 bytes, after stripping a byte-order mark and any HTML comments at the very top, divided by 4. That makes the 8,000-token budget about 32,000 bytes. Exact counts depend on the model’s tokenizer (Anthropic notes its newer tokenizer produces roughly 30 percent more tokens for the same text), so leave headroom rather than aiming at the line.

Sources: LLMSRank — Token budget audit · Claude API docs — Token counting

What happens to the Token budget score past 8,000 tokens?
It falls linearly as 1 − (tokens − 8,000) ÷ 34,000, so 15,000 tokens scores about 0.79 and 25,000 scores 0.5. Above 25,000 tokens (about 100,000 bytes) it drops to a flat 0.3, however large the file.

Sources: LLMSRank — Token budget audit

Does an Optional section still protect a large file?
No. Rubric v1.1 gave a large file with an ## Optional split 0.85; rubric v2.0 scores the budget alone, because llms.txt v2 says “the file itself stays small enough to fit in context” and treats Optional only as a convention for skippable links. Trimming the file is the only fix.

Sources: llmstxt.org — The /llms.txt file, v2 · LLMSRank — Token budget audit

Should detailed content go in llms.txt itself or behind its links?
Behind the links. The proposal is explicit: “The detail lives behind the links, and is fetched only when needed.” Keep the file to a summary, short guidance and described links, and move long reference material into linked markdown pages, which also count toward LLM-friendly links.

Sources: llmstxt.org — The /llms.txt file, v2 · LLMSRank — Token budget audit

How large are real llms.txt files?
In our research, all 187 sites in our corpus (captured 2026-10-10, scored with rubric v2.1) were scored on it, and 149 (79.7%) stayed within 8,000 tokens for full marks; the mean was 0.922. netlify.com, aflac.com, taylormorrison.com, expedia.com and tavily.com are examples.

Sources: LLMSRank — Method · LLMSRank — Token budget audit