llms.txt, robots.txt and sitemap.xml
robots.txt, sitemap.xml and llms.txt are all plain files at predictable URLs, and they are easy to confuse. They do different jobs: robots.txt says what automated access is acceptable, a sitemap lists every indexable page for search engines, and llms.txt picks out what an AI agent should read and describes it. None replaces the others. This page compares them, quotes the llms.txt proposal on how it fits alongside the other two, and shows how they work together.
At a glance
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Says which paths automated clients may fetch | Lists the pages available for indexing | Gives agents a curated, described guide to the content worth reading |
| Who reads it | Crawlers: search engines, AI crawlers, other bots | Search engines | AI agents, on demand, while helping a user |
| Location | /robots.txt at the root of the host only | Usually /sitemap.xml; can live elsewhere and be declared with a Sitemap: line in robots.txt | /llms.txt at the root, or at any path, covering the pages under it |
| Format | Plain text: User-agent, Allow and Disallow rules | XML: a list of URLs with optional lastmod, or an index of sitemaps | Markdown: an H1, optional summary and notes, H2 lists of links with descriptions |
| Completeness | Rules, not a page list | Every indexable page | A selection, and can link to other sites |
| What it controls | Crawler access, for crawlers that honour it | Nothing; it is a list of hints | Nothing; it grants and denies no access |
| Defined by | RFC 9309 (Robots Exclusion Protocol) | The sitemaps.org protocol | The llmstxt.org proposal |
How the proposal positions llms.txt
llms.txt is designed to coexist with current web standards. While sitemaps list all pages for search engines, `llms.txt` offers a curated overview for LLMs. It can complement robots.txt by providing context for allowed content. The file can also reference structured data markup used on the site, helping LLMs understand how to interpret this information in context.
The naming follows the same pattern: “The approach of using a standard filename follows /robots.txt and /sitemap.xml at the site root.” Unlike robots.txt, though, an llms.txt can also sit at a path, and the proposal chose that over RFC 8615’s /.well-known/ prefix because many authors control only a path on a shared host.
Why a sitemap is not a substitute
sitemap.xml is a list of all the indexable human-readable information available on a site. This isn’t a substitute for `llms.txt` since it:
- “Often won’t have the LLM-readable versions of pages listed”
- “Doesn’t include URLs to external sites, even though they might be helpful to understand the information”
- “Will generally cover documents that in aggregate will be too large to fit in an LLM context window, and will include a lot of information that isn’t necessary to understand the site.”
The corollary is that an llms.txt which simply copies the sitemap misses the point. LLMSRank’s rubric has an audit for exactly this, Curation, not a sitemap: on sites with 50 or more sitemap pages, a file that links to most of the sitemap scores lower than one that selects. That is our rubric’s judgement; the proposal describes the difference but sets no threshold.
Inference, not training
The proposal draws the line between robots.txt and llms.txt by when and why each is read:
robots.txt and `llms.txt` have different purposes. robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user. Our expectation was that llms.txt would mainly be useful for *inference* rather than *training*, and that is how it has been used, though training runs could take advantage of the information too.
So llms.txt is not a way to opt in to, or out of, AI training. If you want to control which crawlers fetch your site, that is robots.txt’s job (and, for clients that ignore it, your server’s or CDN’s). llms.txt only shapes what a reader that does arrive finds first.
How they work together
- robots.txt has to let agents fetch it. The proposal doesn’t say how robots.txt rules apply to llms.txt, but an agent that honours robots.txt and is disallowed from
/llms.txtwon’t read it. In our July 2026 study, 3 sites published an llms.txt while blocking AI crawlers from fetching it. - Be explicit about AI crawlers. Only 33 of 148 robots.txt files in our study named any AI crawler. Naming the ones you welcome makes your intent unambiguous. This is LLMSRank’s recommendation (Named AI crawlers), not the proposal’s.
- Keep the sitemap complete and llms.txt selective. The sitemap is still how search engines find every page. llms.txt is the short list, in your order, with a line on what each page answers.
- Point llms.txt at markdown. A sitemap lists HTML pages; llms.txt links should go to LLM-friendly content, such as markdown versions of those pages.
- Keep dates honest. A sitemap’s
lastmodand a date line in llms.txt should agree with when pages really changed. LLMSRank’s rubric (Sitemap freshness) checks that sitemap dates are plausible.
Frequently asked questions
Does llms.txt replace my sitemap?
Does llms.txt control what AI can crawl or train on?
Should I reference llms.txt from robots.txt?
rel="describedby" links.Can robots.txt stop agents reading my llms.txt?
/llms.txt, they won’t fetch it. Three sites in our July 2026 study had exactly this problem.Where does each file go?
/sitemap.xml, or anywhere declared in robots.txt. llms.txt at /llms.txt or at any path, where it covers the pages under that path.