# robots.txt AI crawler builder

> Choose which AI crawlers may fetch your site and get a robots.txt with a line diff, a before and after table per crawler, and your llms.txt kept fetchable.

This is the markdown version of https://llmsrank.dev/tools/robots-txt. The tool is beta and free.

You paste your robots.txt, or fetch the site’s current one, then pick a preset or set each crawler to allow or block. Your file is kept as it is; the builder adds or replaces one block between `# BEGIN AI crawlers` and `# END AI crawlers`. In every group that blocks, `Allow: /llms.txt` (and any llms.txt subpath you add) is kept, because blocking an AI reader from your llms.txt caps the AI crawler access audit. The result comes with a line diff against your file, a table of what each crawler may fetch before and after, and lint findings for both files. Everything runs in the browser; only the optional fetch of your current robots.txt and the optional draft score go through our server.

## Diff, check again and score a draft

- **robots.txt.** The proposed file is shown against your live one as a line diff, with Copy and Download for the proposed file. After you publish, “Check again” fetches only your live robots.txt again (through our server, as LLMSRank-User; it counts against the 100 robots.txt fetches a day) and says what changed since the last fetch and whether it now matches.

“Score this draft” re-scores your site’s latest scan with the draft in place of the live file and shows the scan’s score with and without it, any cap, and every audit that would change. Nothing is published and the draft is not fetched; it is sent to our server only to be scored and is not kept. Audits that depend on pages the scan fetched for your live file’s links are kept from the scan and named. Limits: robots.txt 500 KiB; 100 drafts a day, shared with the robots.txt fetch.

## What robots.txt can and can’t do

- robots.txt is a request, not enforcement. Crawlers that follow it do so voluntarily.
- These vendors say their user-initiated fetchers may not follow robots.txt: OpenAI ChatGPT-User, Perplexity Perplexity-User, Meta Meta-ExternalFetcher, Amazon Amzn-User and Google Google-Agent. Anthropic says Claude-User follows it.
- Blocking a training crawler applies to future crawls. None of the vendor pages cited here say it removes content already collected or models already trained.
- Google-Extended and Applebot-Extended don’t crawl. They tell Google and Apple how to use what their other crawlers fetch, and both say blocking them doesn’t affect search.
- Enforcement, such as a firewall or bot-management rule, is out of scope for this tool.
- IETF AI Preferences (Content-Usage) is still a draft. Shown for information only; the builder doesn’t add it to your file.

## Presets

Each file below is what the builder produces for the preset on an empty robots.txt. Add it to the end of your own file.

### Stay in AI answers, opt out of training

Blocks training crawlers, the Google-Extended and Applebot-Extended control tokens and retired tokens. Allows AI search crawlers, user-initiated fetchers, ad checks and crawls you request. Your llms.txt stays allowed.

```txt
# BEGIN AI crawlers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: DuckAssistBot
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: OAI-AdsBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Allow: /llms.txt
Disallow: /
# END AI crawlers

```

- robots.txt is a request, not enforcement. Crawlers that follow it do so voluntarily.
- Blocking a training crawler applies to future crawls. None of the vendor pages cited here say it removes content already collected or models already trained.
- Google-Extended also covers grounding in Gemini Apps, so blocking it may keep your pages out of grounded Gemini answers. Google says Google Search is unaffected.
- Google-Extended and Applebot-Extended don’t crawl. They tell Google and Apple how to use what their other crawlers fetch, and both say blocking them doesn’t affect search.

### Block all AI

Blocks every AI crawler and control token listed here from the whole site, except your llms.txt.

```txt
# BEGIN AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: OAI-AdsBot
Allow: /llms.txt
Disallow: /
# END AI crawlers

```

- robots.txt is a request, not enforcement. Crawlers that follow it do so voluntarily.
- Your llms.txt stays allowed in every group that blocks. Blocking the whole site without that line caps your LLMSRank score at 25. No major AI vendor documents reading llms.txt today; keeping it allowed means any tool a person points at it can still read it.
- These vendors say their user-initiated fetchers may not follow robots.txt: OpenAI ChatGPT-User, Perplexity Perplexity-User, Meta Meta-ExternalFetcher, Amazon Amzn-User and Google Google-Agent. Anthropic says Claude-User follows it.
- Blocking a search crawler takes you out of that product’s AI answers. OpenAI: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”.
- Blocking a training crawler applies to future crawls. None of the vendor pages cited here say it removes content already collected or models already trained.

### Allow all AI

Allows every AI crawler listed here, training crawlers included.

```txt
# BEGIN AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: OAI-AdsBot
Allow: /
# END AI crawlers

```

- robots.txt is a request, not enforcement. Crawlers that follow it do so voluntarily.
- These vendors say their user-initiated fetchers may not follow robots.txt: OpenAI ChatGPT-User, Perplexity Perplexity-User, Meta Meta-ExternalFetcher, Amazon Amzn-User and Google Google-Agent. Anthropic says Claude-User follows it.

## AI crawler tokens

Grouped by vendor, with what the vendor says on its own page. “Builder only” tokens are ones the vendor documents but our audits do not count yet.

### OpenAI

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `GPTBot` | Training | Vendor says it follows robots.txt | “Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.” | [source](https://developers.openai.com/api/docs/bots), checked 2026-09-29 |
| `OAI-SearchBot` | AI search | Vendor says it follows robots.txt | “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” | [source](https://developers.openai.com/api/docs/bots), checked 2026-09-29 |
| `ChatGPT-User` | User-initiated fetch | Vendor says it may not follow robots.txt | “Because these actions are initiated by a user, robots.txt rules may not apply.” | [source](https://developers.openai.com/api/docs/bots), checked 2026-09-29 |
| `OAI-AdsBot` | Ads | No vendor statement found | OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models. (OpenAI’s page says nothing about robots.txt for it.) Builder only. Documented by the vendor, not yet counted by our rubric. | [source](https://developers.openai.com/api/docs/bots), checked 2026-09-29 |

### Anthropic

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `ClaudeBot` | Training | Vendor says it follows robots.txt | “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” | [source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked 2026-09-29 |
| `Claude-SearchBot` | AI search | Vendor says it follows robots.txt | “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” | [source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked 2026-09-29 |
| `Claude-User` | User-initiated fetch | Vendor says it follows robots.txt | “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query | [source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked 2026-09-29 |
| `anthropic-ai` | Retired token | No vendor statement found | Not on Anthropic’s current crawler page; still common in older robots.txt files. | [source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked 2026-09-29 |

### Perplexity

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `PerplexityBot` | AI search | Vendor says it follows robots.txt | “To ensure your site appears in search results, we recommend allowing PerplexityBot in your site's robots.txt file” | [source](https://docs.perplexity.ai/guides/bots), checked 2026-09-29 |
| `Perplexity-User` | User-initiated fetch | Vendor says it may not follow robots.txt | “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” | [source](https://docs.perplexity.ai/guides/bots), checked 2026-09-29 |

### Google

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `Google-Extended` | Control token (does not crawl) | Not a crawler: a robots.txt token only | “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” Controls use for training Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no separate crawler. | [source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), checked 2026-09-29 |
| `Google-Agent` | User-initiated fetch | Vendor says it may not follow robots.txt | “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.” Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. Builder only. Documented by the vendor, not yet counted by our rubric. | [source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), checked 2026-09-29 |
| `Google-CloudVertexBot` | Crawl the site owner requested | Vendor says it follows robots.txt | “They always obey robots.txt rules when crawling automatically.” Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products. Builder only. Documented by the vendor, not yet counted by our rubric. | [source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), checked 2026-09-29 |

### Apple

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `Applebot-Extended` | Control token (does not crawl) | Not a crawler: a robots.txt token only | “Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results.” Opts out of training Apple’s foundation models. | [source](https://support.apple.com/en-us/119829), checked 2026-09-29 |

### Meta

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `Meta-ExternalAgent` | Training | Vendor says it follows robots.txt | “In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.” | [source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), checked 2026-09-29 |
| `Meta-ExternalFetcher` | User-initiated fetch | Vendor says it may not follow robots.txt | “The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user.” | [source](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), checked 2026-09-29 |

### Amazon

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `Amazonbot` | Training | Vendor says it follows robots.txt | “Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives.” | [source](https://developer.amazon.com/amazonbot), checked 2026-09-29 |
| `Amzn-SearchBot` | AI search | Vendor says it follows robots.txt | “Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives.” | [source](https://developer.amazon.com/amazonbot), checked 2026-09-29 |
| `Amzn-User` | User-initiated fetch | Vendor says it may not follow robots.txt | “Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives.” | [source](https://developer.amazon.com/amazonbot), checked 2026-09-29 |

### Mistral AI

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `MistralAI-User` | User-initiated fetch | Vendor says it follows robots.txt | “MistralAI-User governs which sites these user requests can be made to.” | [source](https://docs.mistral.ai/robots/), checked 2026-09-29 |
| `MistralAI-Index` | AI search | Vendor says it follows robots.txt | “To help webmasters manage how their sites and content interact with AI, Mistral uses specific robots.txt tags.” | [source](https://docs.mistral.ai/robots/), checked 2026-09-29 |
| `MistralAI-Training` | Training | Vendor says it follows robots.txt | “Webmasters can disallow this user agent in their robots.txt file.” | [source](https://docs.mistral.ai/robots/), checked 2026-09-29 |

### DuckDuckGo

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `DuckAssistBot` | AI search | Vendor says it follows robots.txt | “Once you’ve updated the file to disallow the DuckAssistBot user agent, the change will take effect after 72 hours and DuckAssistBot will stop crawling your site.” | [source](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), checked 2026-09-29 |

### Common Crawl

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `CCBot` | Training | Vendor says it follows robots.txt | “To prevent Common Crawl from crawling your website, include the following in your robots.txt” | [source](https://commoncrawl.org/ccbot), checked 2026-09-29 |

### ByteDance

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `Bytespider` | Training | No vendor statement found | no statement found | no statement found |

### Cohere

| Token | Purpose | Robots.txt | Vendor’s words | Source |
| --- | --- | --- | --- | --- |
| `cohere-ai` | Retired token | No vendor statement found | no statement found | no statement found |