robots.txt AI crawler builder

Say which AI crawlers may fetch your site and which may not. Paste your robots.txt or fetch it, pick a preset or set each crawler by hand, and copy the result. Everything runs in your browser; only the optional fetch of your current file goes through our server.

Training, AI search and user-initiated fetching are different things with different tokens, and Google-Extended and Applebot-Extended don’t crawl at all. The table below says which is which, in each vendor’s own words. Your file is kept as it is; the builder only adds or replaces a block between # BEGIN AI crawlers and # END AI crawlers. This is the tool behind the AI crawler access and named AI crawlers audits.

1. Your current robots.txt

Domain to fetch from

Or paste it here (leave empty for a new file)

2. Choose what each AI crawler may fetch

Start from a preset, then change any row. “Leave to *” adds nothing for that token.

Token	Purpose	Does the vendor say it follows robots.txt?
OpenAI
`GPTBot`	Training	Vendor says it follows robots.txt Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Source, checked 2026-09-29
`OAI-SearchBot`	AI search	Vendor says it follows robots.txt Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. Source, checked 2026-09-29
`ChatGPT-User`	User-initiated fetch	Vendor says it may not follow robots.txt Because these actions are initiated by a user, robots.txt rules may not apply. Source, checked 2026-09-29
`OAI-AdsBot`Builder only	Ads	No vendor statement found OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models. (OpenAI’s page says nothing about robots.txt for it.) Source, checked 2026-09-29 Documented by the vendor, not yet counted by our rubric.
Anthropic
`ClaudeBot`	Training	Vendor says it follows robots.txt Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt. Source, checked 2026-09-29
`Claude-SearchBot`	AI search	Vendor says it follows robots.txt Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt. Source, checked 2026-09-29
`Claude-User`	User-initiated fetch	Vendor says it follows robots.txt Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query Source, checked 2026-09-29
`anthropic-ai`	Retired token	No vendor statement found Not on Anthropic’s current crawler page; still common in older robots.txt files. Source, checked 2026-09-29
Perplexity
`PerplexityBot`	AI search	Vendor says it follows robots.txt To ensure your site appears in search results, we recommend allowing PerplexityBot in your site's robots.txt file Source, checked 2026-09-29
`Perplexity-User`	User-initiated fetch	Vendor says it may not follow robots.txt Since a user requested the fetch, this fetcher generally ignores robots.txt rules. Source, checked 2026-09-29
Google
`Google-Extended`	Control token (does not crawl)	Not a crawler: a robots.txt token only Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. Controls use for training Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no separate crawler. Source, checked 2026-09-29
`Google-Agent`Builder only	User-initiated fetch	Vendor says it may not follow robots.txt Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. Source, checked 2026-09-29 Documented by the vendor, not yet counted by our rubric.
`Google-CloudVertexBot`Builder only	Crawl the site owner requested	Vendor says it follows robots.txt They always obey robots.txt rules when crawling automatically. Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products. Source, checked 2026-09-29 Documented by the vendor, not yet counted by our rubric.
Apple
`Applebot-Extended`	Control token (does not crawl)	Not a crawler: a robots.txt token only Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Opts out of training Apple’s foundation models. Source, checked 2026-09-29
Meta
`Meta-ExternalAgent`	Training	Vendor says it follows robots.txt In order to block these crawlers, add a disallow for the relevant crawler to robots.txt. Source, checked 2026-09-29
`Meta-ExternalFetcher`	User-initiated fetch	Vendor says it may not follow robots.txt The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user. Source, checked 2026-09-29
Amazon
`Amazonbot`	Training	Vendor says it follows robots.txt Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives. Source, checked 2026-09-29
`Amzn-SearchBot`	AI search	Vendor says it follows robots.txt Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives. Source, checked 2026-09-29
`Amzn-User`	User-initiated fetch	Vendor says it may not follow robots.txt Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives. Source, checked 2026-09-29
Mistral AI
`MistralAI-User`	User-initiated fetch	Vendor says it follows robots.txt MistralAI-User governs which sites these user requests can be made to. Source, checked 2026-09-29
`MistralAI-Index`	AI search	Vendor says it follows robots.txt To help webmasters manage how their sites and content interact with AI, Mistral uses specific robots.txt tags. Source, checked 2026-09-29
`MistralAI-Training`	Training	Vendor says it follows robots.txt Webmasters can disallow this user agent in their robots.txt file. Source, checked 2026-09-29
DuckDuckGo
`DuckAssistBot`	AI search	Vendor says it follows robots.txt Once you’ve updated the file to disallow the DuckAssistBot user agent, the change will take effect after 72 hours and DuckAssistBot will stop crawling your site. Source, checked 2026-09-29
Common Crawl
`CCBot`	Training	Vendor says it follows robots.txt To prevent Common Crawl from crawling your website, include the following in your robots.txt Source, checked 2026-09-29
ByteDance
`Bytespider`	Training	No vendor statement found
Cohere
`cohere-ai`	Retired token	No vendor statement found

3. Keep your llms.txt fetchable

Your llms.txt stays allowed in every group that blocks. Blocking the whole site without that line caps your LLMSRank score at 25. No major AI vendor documents reading llms.txt today; keeping it allowed means any tool a person points at it can still read it.

/llms.txt

Another llms.txt path

4. Optional: state your usage terms

Off unless you choose something here. These lines state terms; they do not block anything, and we have not verified which AI vendors honour them. A Content-Signal line (Cloudflare’s Content Signals Policy) says yes or no per use; a License line points to a machine-readable licence (RSL). Both go into every group the builder writes and a User-agent: * group. The builder warns when they contradict your rules, for example ai-train=no while a training crawler is allowed.

search: a search index and results with links and short excerpts ai-input: content used for AI answers (grounding, retrieval) ai-train: training or fine-tuning AI models

Licence URL (RSL, optional)

5. Your robots.txt

Choose a preset or a rule above and the file appears here.

What robots.txt can and can’t do

robots.txt is a request, not enforcement. Crawlers that follow it do so voluntarily.
These vendors say their user-initiated fetchers may not follow robots.txt: OpenAI ChatGPT-User, Perplexity Perplexity-User, Meta Meta-ExternalFetcher, Amazon Amzn-User and Google Google-Agent. Anthropic says Claude-User follows it.
Blocking a training crawler applies to future crawls. None of the vendor pages cited here say it removes content already collected or models already trained.
Google-Extended and Applebot-Extended don’t crawl. They tell Google and Apple how to use what their other crawlers fetch, and both say blocking them doesn’t affect search.
Enforcement, such as a firewall or bot-management rule, is out of scope for this tool.
IETF AI Preferences (Content-Usage) is still a draft. Shown for information only; the builder doesn’t add it to your file.

Preset files, ready to copy

Each file below is what the builder produces for a preset on an empty robots.txt. Add it to the end of your own file. Every group that blocks keeps your /llms.txt allowed.

Stay in AI answers, opt out of training

Blocks training crawlers, the Google-Extended and Applebot-Extended control tokens and retired tokens. Allows AI search crawlers, user-initiated fetchers, ad checks and crawls you request. Your llms.txt stays allowed.

# BEGIN AI crawlers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: DuckAssistBot
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: OAI-AdsBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Allow: /llms.txt
Disallow: /
# END AI crawlers

Block all AI

Blocks every AI crawler and control token listed here from the whole site, except your llms.txt.

# BEGIN AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: OAI-AdsBot
Allow: /llms.txt
Disallow: /
# END AI crawlers

Allow all AI

Allows every AI crawler listed here, training crawlers included.

# BEGIN AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: OAI-AdsBot
Allow: /
# END AI crawlers