Skip to content
LLMSRank
Docs menu

AI crawler access

AI crawlers permitted to fetch it

In short: Make sure robots.txt blocks none of the seven user agents this audit checks from your llms.txt (add Allow: /llms.txt to any Disallow: / group), and name at least one AI crawler.

  • Category: Retrievability & Access · weight 25 of 100 within the category
  • Retrievability & Access is worth 15 in every profile (points of 100, by scoring profile)
  • Audit ID R5 (as shown on reports) · rubric v2.1

What it checks

Whether robots.txt lets the major AI crawlers fetch your llms.txt.

Why it matters

A file agents can’t fetch can’t be used by any agent or tool you point at it. Blocking AI crawlers from it in robots.txt is the most expensive mistake in the rubric: it caps the whole score at 25.

When it applies

Applies whenever we archived a readable robots.txt for your site, or robots.txt answered with a 4xx such as 404. Under RFC 9309 a 4xx means there are no rules and every crawler may fetch everything, so R5 scores 0.5, the same as a robots.txt that names no AI crawler and blocks none. If robots.txt timed out, answered a 5xx or a 429, or we couldn’t read it, R5 is Unverified: removed from scoring and flagged for you to confirm, never scored as a zero.

An audit that doesn't apply, or that we couldn't verify, is removed from your score and its weight shared among the Retrievability & Access audits that do apply. It is never scored as a zero.

How it's scored

1.0 names at least one AI crawler and blocks none of the readers · 0.5 names none and blocks none · 0 any reader blocked (which also caps the score at 25). Blocked means robots.txt, read per RFC 9309, doesn’t allow that reader to fetch your llms.txt. “Readers” is our term for the seven user agents this audit checks, not a list of agents known to read llms.txt: GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended (a control token, not a crawler; we check the rules written for it), OAI-SearchBot and anthropic-ai.

How to fix

  1. Allow /llms.txt for each AI crawler you want to be able to fetch it. The Named AI crawlers audit page (/docs/audits/named-ai-crawlers) has the full, vendor-documented list of tokens, grouped by vendor and purpose.
  2. Remove any Disallow: / that applies to those agents, including one under User-agent: * when they have no group of their own. If you must keep Disallow: /, add Allow: /llms.txt to the same group: the longer rule wins, so the file stays readable.
  3. Re-check after CDN or bot-management changes, which often rewrite robots.txt.

Example

# Let AI crawlers and agents fetch it. Full vendor list: /docs/audits/named-ai-crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

Frequently asked questions

If my robots.txt has Disallow: / for GPTBot, does adding Allow: /llms.txt in the same group work?
Yes. RFC 9309 says the most specific (longest) matching rule wins, so Allow: /llms.txt beats Disallow: / for that path and we don’t count the crawler as blocked. The carve-out has to match the file that covers your URL: if that is /docs/llms.txt, allow /docs/llms.txt too, and watch for wildcard rules such as Disallow: /*.txt$ that also match it.

Sources: RFC 9309: Robots Exclusion Protocol · AI crawler access

I have User-agent: * with Disallow: /, plus a GPTBot group that allows everything. Am I blocking AI crawlers?
GPTBot is fine, but every other user agent this audit checks falls back to the * group when it has no group of its own, as RFC 9309 requires, so ClaudeBot, Claude-User, PerplexityBot, Google-Extended, OAI-SearchBot and anthropic-ai are all blocked. Any one of them blocked scores this audit 0 and caps the total at 25; give each a group, or add Allow: /llms.txt under *.

Sources: RFC 9309: Robots Exclusion Protocol · AI crawler access

Can I block AI training crawlers and still get full marks?
Partly. Only the seven user agents this audit checks (its “readers”, a scoring term, not a list of agents known to read llms.txt) count as blocked: GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, OAI-SearchBot and anthropic-ai. Blocking others such as CCBot or Bytespider costs nothing, and naming them still counts as naming an AI crawler; GPTBot, ClaudeBot and Google-Extended are training controls, so if you disallow them sitewide, add Allow: /llms.txt in their groups (OpenAI documents that each of its crawler settings is independent).

Sources: OpenAI: crawlers · Google: common crawlers (Google-Extended) · Named AI crawlers

What if my robots.txt returns 404, or your scan couldn’t read it?
A 404, or any other 4xx except 429, means there is no robots.txt, and RFC 9309 says crawlers may then fetch everything. So nothing is blocked and no AI crawler is named: the audit scores 0.5, the same as a robots.txt that relies on User-agent: *, with no cap. If robots.txt timed out, answered a 5xx or a 429 (a rate limit, not an answer about the file), or we couldn’t read it, the audit is Unverified: removed from scoring and flagged for you to check, never scored as zero. Publishing a robots.txt that names the AI crawlers you allow earns full marks.

Sources: RFC 9309 §2.3.1: Access results · AI crawler access

In your research, how many sites let AI crawlers fetch their llms.txt?
Of the 148 tracked sites where we could read robots.txt (8 more were Unverified), 20.3% (30) earn full marks by naming an AI crawler and blocking none of the user agents this audit checks, and 2.7% (4) block one of them, for a mean score of 0.588. The other 114 block none of them but name no AI crawler at all, so they score 0.5. Among those with full marks are netlify.com, zapier.com, retool.com, jasper.ai and cloudflare.com.

Sources: LLMSRank method · AI crawler access