Skip to content
LLMSRank
Docs menu

Clean fetch

Clean fetch — 200, no auth, ≤1 same-host redirect

In short: Serve /llms.txt with a plain 200 to anonymous requests: no login, bot challenge or WAF block and at most one same-host redirect (redirects are not penalised yet).

  • Category: Retrievability & Access · weight 20 of 100 within the category
  • Retrievability & Access is worth 15 in every profile (points of 100, by scoring profile)
  • Audit ID R2 (as shown on reports) · rubric v2.1

What it checks

Whether the file fetches cleanly: HTTP 200, no auth, at most one same-host redirect.

Why it matters

Most agent fetchers make one request and give up on logins, challenges or long redirect chains. A file behind any of them is effectively absent.

When it applies

Always, for any llms.txt the scan fetched. A fetch that fails is never scored as a zero: a 401 (login required), 403 or 429 is reported as the scan being turned away, a timeout or a 5xx as Unverified, and a login page served with 200 as a web page answering instead of a file.

An audit that doesn't apply, or that we couldn't verify, is removed from your score and its weight shared among the Retrievability & Access audits that do apply. It is never scored as a zero.

How it's scored

1.0 for every file the scan could fetch. The scanner follows redirects like most HTTP clients, so a file reached through a redirect scores the same as one served directly.

How to fix

  1. Serve /llms.txt directly with a 200; avoid redirecting it to another host or path.
  2. Exclude the path from bot challenges, WAF rules and login walls.
  3. If you must redirect (for example http → https), keep it to a single same-host hop.

Frequently asked questions

Does redirecting /llms.txt, for example from http to https or to /static/llms.txt, cost me points?
The rubric gives 1.0 for a direct 200, 0.5 for one same-host redirect and 0 for a chain, auth or error. Today the scanner follows redirects like most HTTP clients and scores any file it reaches the same, so this audit is not yet measured precisely; serve the file directly anyway, since some agent fetchers make one request and stop.

Sources: Clean fetch · RFC 9110 §15.4: Redirection 3xx

Why did my scan say we were turned away at the door when the file loads fine in my browser?
The server answered 401 (Unauthorized), 403 (Forbidden) or 429 (Too Many Requests) to our request, so nothing was scored and nothing counts against you. A 403 or 429 is almost always a WAF, bot-management rule or rate limit that treats a non-browser client differently from you; allow the LLMSRank-User user agent (and the AI fetchers you care about) for /llms.txt and scan again. A 401 means the path asks for a login, which your browser session may be satisfying for you; agents have no session, so make the file public.

Sources: RFC 9110 §15.5.2: 401 Unauthorized · RFC 9110 §15.5.4: 403 Forbidden · RFC 6585 §4: 429 Too Many Requests

How do I let bots reach /llms.txt through Cloudflare’s bot protection?
On paid plans, add a WAF custom rule matching http.request.uri.path eq "/llms.txt" with the Skip action so managed rules and Super Bot Fight Mode don’t challenge it. Cloudflare documents that the Free plan’s Bot Fight Mode cannot be skipped, so on that plan you have to turn it off or accept that some fetchers will be challenged.

Sources: Cloudflare: skip rules

What happens if /llms.txt needs a login, times out or returns a 5xx error?
A 401 (login required) is reported as the scan being turned away, like a 403: the file isn’t publicly retrievable, so nothing is scored, and agent fetchers won’t log in either, so make the path public. A timeout, a 5xx or a status we don’t interpret makes the scan Unverified: nothing is scored and nothing is held against the site, so try again later. A login page served with 200 is an HTML page, so the scan reports that a web page answered instead of scoring a file.

Sources: RFC 9110: HTTP Semantics · Clean fetch

In your research, how many sites pass the Clean fetch audit?
All 156 tracked sites score 1.0, but only because this audit is not yet measured precisely: every file we could fetch gets the same score, and a fetch that fails is reported as an outcome (blocked, not present, unverified) rather than scored. It says nothing yet about how many sites redirect their llms.txt.

Sources: LLMSRank method · Clean fetch