llms.txt vs AGENTS.md vs robots.txt

They are three files for three readers: robots.txt tells crawlers which paths they may fetch, llms.txt gives AI agents a curated, described map of a site’s content, and AGENTS.md gives coding agents instructions for working in a code repository. None replaces the others, and only robots.txt says anything about access.

Updated 2026-09-29

At a glance

	robots.txt	llms.txt	AGENTS.md
Purpose	Says which paths automated clients may fetch	A curated, described guide to the content worth reading	Instructions for a coding agent: build steps, tests, conventions
Who reads it	Crawlers that honour the protocol: search engines, AI crawlers, other bots	AI agents, on demand, while helping a user (per the proposal); tools you point at it	Coding agents working in the repository; agents.md lists more than 20, from Codex to Cursor and Gemini CLI
Location	`/robots.txt` at the top level of the host only	`/llms.txt` at the root, or at any path, covering the pages under it	The repository root, plus nested files in subprojects; the closest one to the edited file wins
Format	Plain text, UTF-8: `User-agent`, `Allow` and `Disallow` rules	Markdown with a fixed shape: an H1, optional summary and notes, H2 lists of links	Standard markdown, any headings, no required fields
What it controls	Crawler access, for crawlers that honour it; not access authorization	Nothing; it grants and denies no access	Nothing outside the agent’s work; it is guidance
Defined by	RFC 9309 (September 2022)	The llmstxt.org proposal (v2, 2026-08-10)	agents.md, stewarded by the Agentic AI Foundation under the Linux Foundation

For how llms.txt differs from a sitemap, see llms.txt, robots.txt and sitemap.xml.

robots.txt: access rules for crawlers

This document specifies and extends the “Robots Exclusion Protocol” method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.
Source: RFC 9309 — Robots Exclusion Protocol

The file has one place: “The rules MUST be accessible in a file named “/robots.txt” (all lowercase) in the top-level path of the service.” And it is a request, not a lock: “These rules are not a form of access authorization.”

Vendors differ on whether fetches a person asks for follow it. Anthropic says “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt” (article updated 2026-04-07). OpenAI says that for ChatGPT-User, “robots.txt rules may not apply”, and Perplexity says Perplexity-User “generally ignores robots.txt rules” (both pages undated, checked 2026-09-29).

llms.txt: a map of your content for AI agents

We propose adding a `/llms.txt` markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files.
Source: llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)

The proposal says it “can complement robots.txt by providing context for allowed content”: robots.txt decides what may be fetched, llms.txt says what is worth reading. Which AI products actually read the file is a separate question, answered vendor by vendor in Does ChatGPT, Claude, Perplexity or Google read llms.txt?. The format itself is on About llms.txt.

AGENTS.md: instructions for coding agents

A simple, open format for guiding coding agents, used by over 60k open-source projects.
Source: agents.md — AGENTS.md, a simple, open format for guiding coding agents

agents.md positions it next to a README: the README is for people, and AGENTS.md holds “the extra, sometimes detailed context coding agents need: build steps, tests, and conventions.” It has no schema: “AGENTS.md is just standard Markdown. Use any headings you like.” Nested files are allowed, and “The closest AGENTS.md to the edited file wins.”

agents.md describes the file in a code repository. Some sites also publish an AGENTS.md at their web root, with instructions for agents using their product or API; the format doesn’t define that use, and we found no agent documented as fetching a website’s /AGENTS.md. LLMSRank’s Agent files audit checks for one at the root. That is our rubric’s opinion, not part of the agents.md format.

In the llms.txt Index, 11 of the 187 sites (6%) earn full marks on that audit with a substantive root AGENTS.md; 93% score zero. The agent-instructions builder drafts both the llms.txt section and a root AGENTS.md from a short form.

How they work together

robots.txt lets agents reach llms.txt. If you disallow a crawler sitewide but want it to read your llms.txt, add Allow: /llms.txt to its group: under RFC 9309 the longer, more specific rule wins.
llms.txt says what to read. Link your best pages, each with a line on what it answers. If you publish an AGENTS.md, you can link it from llms.txt too.
AGENTS.md says how to act. Build and test commands in a repository; authentication, base URLs and limits if you publish one for agents using your API.
Don’t make one do another’s job. llms.txt can’t block anything, robots.txt can’t describe anything, and AGENTS.md is not a content index.

Frequently asked questions

What is the difference between llms.txt and AGENTS.md?

llms.txt is a curated map of a website’s content for AI agents, at /llms.txt. AGENTS.md is instructions for coding agents working in a code repository, such as build steps, tests and conventions. One tells an agent what to read; the other tells it how to work.

What is the difference between llms.txt and robots.txt?

robots.txt says which paths crawlers may fetch, for crawlers that honour RFC 9309. llms.txt grants and denies nothing; it points agents to the content worth reading.

Can llms.txt block AI crawlers or AI training?

No. It has no access rules. robots.txt is the file crawlers check for access, and RFC 9309 notes its rules “are not a form of access authorization”.

Where does AGENTS.md go?

At the root of your code repository, with optional nested files in subprojects; the closest one to the file being edited wins. agents.md doesn’t define a location on a website.

Do I need all three files?

Most public websites already have a robots.txt. Add llms.txt if you want AI agents to find your best pages quickly, and AGENTS.md if coding agents work in your repositories or on your API. None is required by the others.

Do AI assistants follow robots.txt when a user asks them to open a page?

It depends on the vendor. Anthropic says its bots honour robots.txt; OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores robots.txt rules” (checked 2026-09-29).