Named AI crawlers

robots.txt names AI crawlers explicitly

In short: Give the AI crawlers you have a view on their own User-agent groups in robots.txt, and add Allow: /llms.txt to any group that says Disallow: /.

• Category: Ecosystem Integration · weight 22 of 100 within the category
• Ecosystem Integration is worth 20 in technical, 15 in generic, regulated and commerce (points of 100, by scoring profile)
• Audit ID E1 (as shown on reports) · rubric v2.1

What it checks

Whether robots.txt names AI crawlers explicitly, rather than relying on a catch-all rule.

Why it matters

An explicit rule per AI crawler is an unambiguous statement of intent that crawlers, and the people auditing them, can read. A catch-all Allow: / works, but says nothing about AI.

When it applies

Applies whenever we archived a readable robots.txt for your site, or robots.txt answered with a 4xx such as 404. A missing robots.txt names no AI crawler and, under RFC 9309, blocks nothing, so E1 scores 0.5. If robots.txt timed out, answered a 5xx or a 429, or we couldn’t read it, E1 is Unverified: removed from scoring and flagged for you to confirm.

An audit that doesn't apply, or that we couldn't verify, is removed from your score and its weight shared among the Ecosystem Integration audits that do apply. It is never scored as a zero.

How it's scored

1.0 names at least one AI crawler and blocks none of the readers · 0.5 names none and blocks none · 0.25 a reader is blocked from your llms.txt (readers and blocking as defined on AI crawler access: the seven user agents that audit checks, RFC 9309, User-agent: * included). A crawler is named when a User-agent line’s value is one of the 23 tokens we recognise (the 20 in the example below, plus the legacy anthropic-ai and cohere-ai, and Bytespider), case aside; a trailing # comment and a version suffix (GPTBot/1.0) are ignored.

How to fix

1. Name each AI crawler you have a view on, grouped by vendor as in the example. Several User-agent lines in a row share the rules below them (RFC 9309).
2. Decide per purpose, not per company: training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, MistralAI-Training, CCBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index, DuckAssistBot) and user-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, MistralAI-User). To stay in AI answers but out of training, allow the search and user agents, and give the training ones Disallow: / together with Allow: /llms.txt in the same group. GPTBot, ClaudeBot and Google-Extended are also among the seven user agents AI crawler access checks, so a bare Disallow: / for them blocks your llms.txt and caps the whole score at 25; the longer Allow: /llms.txt rule wins (RFC 9309) and keeps the file readable.
3. Google-Extended and Applebot-Extended are control tokens, not crawlers: Googlebot and Applebot do the fetching, and these only decide whether the content may be used for Gemini or Apple model training. Blocking them does not remove you from Google or Apple search.
4. Keep the groups consistent with AI crawler access: wherever a group says Disallow: /, add Allow: /llms.txt (or the path of the llms.txt that covers your pages, such as /docs/llms.txt) for any crawler you want to be able to fetch it. OpenAI and Perplexity say their user-requested fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt because a person asked for the page, while Anthropic says its bots, Claude-User included, honour it; name them anyway so your intent is on record.

Example

# AI crawlers, grouped by vendor. Consecutive User-agent lines share the rules below them.
# Tokens from each vendor’s documentation, checked 2026-09-28.

# OpenAI: training, ChatGPT search, pages fetched when a user asks
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /

# Anthropic: training, Claude search, pages fetched when a user asks
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Perplexity: search, pages fetched when a user asks
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Control tokens: Googlebot and Applebot fetch; these govern use for Gemini / Apple AI training
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /

# Meta, Amazon, Mistral AI, DuckDuckGo, Common Crawl
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: CCBot
Allow: /

# Everyone else
User-agent: *
Allow: /