# llms.txt, robots.txt and sitemap.xml

robots.txt, sitemap.xml and llms.txt are all plain files at predictable URLs, and they are easy to confuse. They do different jobs: robots.txt says what automated access is acceptable, a sitemap lists every indexable page for search engines, and llms.txt picks out what an AI agent should read and describes it. None replaces the others. This page compares them, quotes the llms.txt proposal on how it fits alongside the other two, and shows how they work together.

Updated 2026-09-29. This is the markdown version of https://llmsrank.dev/learn/llms-txt-robots-txt-sitemap.

## At a glance

|  | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| **Purpose** | Says which paths automated clients may fetch | Lists the pages available for indexing | Gives agents a curated, described guide to the content worth reading |
| **Who reads it** | Crawlers: search engines, AI crawlers, other bots | Search engines | AI agents, on demand, while helping a user |
| **Location** | `/robots.txt` at the root of the host only | Usually `/sitemap.xml`; can live elsewhere and be declared with a `Sitemap:` line in robots.txt | `/llms.txt` at the root, or at any path, covering the pages under it |
| **Format** | Plain text: `User-agent`, `Allow` and `Disallow` rules | XML: a list of URLs with optional `lastmod`, or an index of sitemaps | Markdown: an H1, optional summary and notes, H2 lists of links with descriptions |
| **Completeness** | Rules, not a page list | Every indexable page | A selection, and can link to other sites |
| **What it controls** | Crawler access, for crawlers that honour it | Nothing; it is a list of hints | Nothing; it grants and denies no access |
| **Defined by** | RFC 9309 (Robots Exclusion Protocol) | The sitemaps.org protocol | The llmstxt.org proposal |

## How the proposal positions llms.txt

> llms.txt is designed to coexist with current web standards. While sitemaps list all pages for search engines, `llms.txt` offers a curated overview for LLMs. It can complement robots.txt by providing context for allowed content. The file can also reference structured data markup used on the site, helping LLMs understand how to interpret this information in context.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

The naming follows the same pattern: “The approach of using a standard filename follows `/robots.txt` and `/sitemap.xml` at the site root.” Unlike robots.txt, though, an llms.txt can also sit at a path, and the proposal chose that over RFC 8615’s `/.well-known/` prefix because many authors control only a path on a shared host.

## Why a sitemap is not a substitute

> sitemap.xml is a list of all the indexable human-readable information available on a site. This isn’t a substitute for `llms.txt` since it:
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

- “Often won’t have the LLM-readable versions of pages listed”
- “Doesn’t include URLs to external sites, even though they might be helpful to understand the information”
- “Will generally cover documents that in aggregate will be too large to fit in an LLM context window, and will include a lot of information that isn’t necessary to understand the site.”

The corollary is that an llms.txt which simply copies the sitemap misses the point. LLMSRank’s rubric has an audit for exactly this, [Curation, not a sitemap](https://llmsrank.dev/docs/audits/curation-not-sitemap): on sites with 50 or more sitemap pages, a file that links to most of the sitemap scores lower than one that selects. That is our rubric’s judgement; the proposal describes the difference but sets no threshold.

## Inference, not training

The proposal draws the line between robots.txt and llms.txt by when and why each is read:

> robots.txt and `llms.txt` have different purposes. robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user. Our expectation was that llms.txt would mainly be useful for *inference* rather than *training*, and that is how it has been used, though training runs could take advantage of the information too.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

So llms.txt is not a way to opt in to, or out of, AI training. If you want to control which crawlers fetch your site, that is robots.txt’s job (and, for clients that ignore it, your server’s or CDN’s). llms.txt only shapes what a reader that does arrive finds first.

## How they work together

- **robots.txt has to let agents fetch it.** The proposal doesn’t say how robots.txt rules apply to llms.txt, but an agent that honours robots.txt and is disallowed from `/llms.txt` won’t read it. In our July 2026 study, 3 sites published an llms.txt while blocking AI crawlers from fetching it.
- **Be explicit about AI crawlers.** Only 33 of 148 robots.txt files in our study named any AI crawler. Naming the ones you welcome makes your intent unambiguous. This is LLMSRank’s recommendation ([Named AI crawlers](https://llmsrank.dev/docs/audits/named-ai-crawlers)), not the proposal’s.
- **Keep the sitemap complete and llms.txt selective.** The sitemap is still how search engines find every page. llms.txt is the short list, in your order, with a line on what each page answers.
- **Point llms.txt at markdown.** A sitemap lists HTML pages; llms.txt links should go to LLM-friendly content, such as [markdown versions](https://llmsrank.dev/learn/markdown-pages) of those pages.
- **Keep dates honest.** A sitemap’s `lastmod` and a date line in llms.txt should agree with when pages really changed. LLMSRank’s rubric ([Sitemap freshness](https://llmsrank.dev/docs/audits/sitemap-freshness)) checks that sitemap dates are plausible.

> **Note: What LLMSRank reads.** A scan fetches all three: robots.txt first, then llms.txt, llms-full.txt and AGENTS.md, then the sitemap named in robots.txt or at `/sitemap.xml`. How each feeds the score is in [/docs/method](https://llmsrank.dev/docs/method).

## Frequently asked questions

### Does llms.txt replace my sitemap?

No. A sitemap lists every indexable page for search engines; llms.txt is a curated selection for AI agents. The proposal gives three reasons a sitemap is no substitute, and none for dropping the sitemap.

### Does llms.txt control what AI can crawl or train on?

No. It grants and denies nothing. Access rules belong in robots.txt, and the proposal expects llms.txt to be used mainly at inference time, when an agent is helping a user.

### Should I reference llms.txt from robots.txt?

The proposal doesn’t define any robots.txt directive for llms.txt. What matters is that your robots.txt allows the agents you want to fetch it; for discovery from your pages, v2 recommends `rel="describedby"` links.

### Can robots.txt stop agents reading my llms.txt?

Yes, for agents that honour robots.txt: if their user agent is disallowed from `/llms.txt`, they won’t fetch it. Three sites in our July 2026 study had exactly this problem.

### Where does each file go?

robots.txt only at the root of the host. A sitemap usually at `/sitemap.xml`, or anywhere declared in robots.txt. llms.txt at `/llms.txt` or at any path, where it covers the pages under that path.

### Can llms.txt link to pages that are not in my sitemap?

Yes. The proposal counts leaving out external URLs “even though they might be helpful” as a weakness of sitemaps, and its own example links to other sites.

## Sources

- [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)
- [llmstxt.org — The /llms.txt file, v2, markdown version](https://llmstxt.org/index.md)
- [RFC 9309 — Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309)
- [sitemaps.org — Sitemaps XML format](https://www.sitemaps.org/protocol.html)
- [RFC 8615 — Well-Known Uniform Resource Identifiers](https://datatracker.ietf.org/doc/html/rfc8615)

## Related

- [About llms.txt](https://llmsrank.dev/learn/llms-txt.md)
- [Writing an llms.txt file](https://llmsrank.dev/learn/writing-llms-txt.md)
- [Markdown page versions and link relations](https://llmsrank.dev/learn/markdown-pages.md)