# About llms.txt

llms.txt is a proposed markdown file that a website publishes to give AI agents a short, curated guide to its content: a title, a summary, and lists of links to pages worth reading, ideally in markdown. It was proposed by Jeremy Howard in September 2024 and revised as v2 in August 2026. This page explains the proposal as its own text does, quotes it where the wording matters, and says plainly where it is silent.

Updated 2026-09-29. This is the markdown version of https://llmsrank.dev/learn/llms-txt.

## What it is

> We propose adding a `/llms.txt` markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

It is one text file, written in markdown so that a language model can read it directly, but with a fixed structure so that ordinary code can parse it too:

> llms.txt markdown is human and LLM readable, but is also in a precise format allowing fixed processing methods (i.e. classical programming techniques such as parsers and regex).
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

The proposal pairs the file with a second idea: pages that agents might need should have a clean markdown version at a predictable URL. That convention, and how pages advertise it, has [its own page](https://llmsrank.dev/learn/markdown-pages).

## Why it exists

The proposal’s background starts from how agents already use the web: a coding agent fetching a library’s documentation, a chat assistant with search reading a product’s pages. Its argument is that web pages are an expensive way to feed them:

> But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money. Agents are best served by concise, expert-level information gathered in a single, accessible location.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

Software documentation is where the file is used most, but the proposal names other fits too:

> llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials. The same structure works anywhere agents need a guided path into a site’s content: a business outlining its structure and policies, a personal site answering questions about someone’s CV, or a school providing access to course information.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

## Where it lives

> The llms.txt file spec is for files named `llms.txt`, at the root path `/llms.txt` of a website or at any subpath (e.g. `/docs/llms.txt`). A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

- **The name is fixed:** `llms.txt`.
- **The place is flexible:** the site root (`https://example.com/llms.txt`) or any path (`https://example.com/docs/llms.txt`).
- **A file covers everything under its path.** `/docs/llms.txt` covers every page in `/docs/`; `/llms.txt` covers the whole site.
- **The most specific file wins.** A page at `/docs/api/auth` covered by both `/llms.txt` and `/docs/llms.txt` should be read with `/docs/llms.txt`.

Subpath files are what let people who control only part of a host take part: a GitHub Pages project site, say, or a docs section run by a different team. The proposal doesn’t say anything about subdomains. A file at `docs.example.com/llms.txt` covers paths on that host; nothing in the text makes it cover `example.com`.

> **Note: What LLMSRank checks today.** Our scan follows v2: enter a page URL and we look for the most specific llms.txt that covers it, from that page’s directory up to the root, and score that file. A file at `/docs/llms.txt` covering the page you entered earns full credit on audit [Covering llms.txt](https://llmsrank.dev/docs/audits/covering-llms-txt), and the report names the file it used.

## The format

A file following the proposal contains these elements, as markdown, in this order:

| # | Element | Required? | What the proposal says |
|---|---|---|---|
| 1 | Byte-order mark (BOM) | No | “An optional byte-order mark (BOM)” |
| 2 | H1: `# Name` | **Yes, the only required element** | “An H1 with the name of the project or site. This is the only required section” |
| 3 | Blockquote: `> Summary` | No | “A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file” |
| 4 | Body: paragraphs, lists | No | “Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information about the project and how to interpret the provided files” |
| 5 | H2 sections of file lists | No | “Zero or more markdown sections delimited by H2 headers, containing “file lists” of URLs where further detail is available” |

Each entry in a file list is a markdown list item with a required link and optional notes after a colon:

> Each “file list” is a markdown list, containing a required markdown hyperlink `[name](url)`, then optionally a `:` and notes about the file.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

The proposal’s own mock example:

```markdown
# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)
```

H2 section names are free text: `## Docs`, `## Services`, `## Policies`. One name has a convention attached:

> The “Optional” section is used, by convention, for secondary information: links an agent can skip when a shorter context is needed.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

In v1, `## Optional` had a special meaning for tools that expanded a file into one big context. v2 removed those tools from the proposal, and the special meaning with them:

> Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics.
>
> Source: [llmstxt.org — Changes: v2 (August 2026)](https://llmstxt.org/changes.html)

The proposal doesn’t say which Content-Type to serve the file with, how large it may be beyond “small enough to fit in context”, whether links must be absolute, or whether to date the file. Those are choices, and where LLMSRank’s rubric has an opinion on them it is marked as ours on the [writing guide](https://llmsrank.dev/learn/writing-llms-txt).

## How agents use it

v1 made no recommendation about how the file should be processed. v2 states the expectation:

> Agents are expected to view or search `llms.txt` to find the information they need, then follow the relevant links. The links in an llms.txt file should therefore point to LLM-friendly content, such as the markdown versions of pages described above. The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

Two consequences for authors: the file is an index, not the content, so keep it short; and each link should lead somewhere an agent can read cheaply, which usually means a markdown version of the page.

The proposal also says the file is read on demand rather than harvested for training: “Our expectation was that llms.txt would mainly be useful for *inference* rather than *training*, and that is how it has been used, though training runs could take advantage of the information too.”

## Who publishes it

The v2 background describes “thousands of sites” publishing a file, and names these:

- **AI labs, for their own developer docs:** [OpenAI](https://developers.openai.com/llms.txt), [Anthropic](https://docs.anthropic.com/llms.txt) and [Gemini](https://ai.google.dev/gemini-api/docs/llms.txt).
- **Chrome’s Lighthouse,** which audits whether a site’s file can be fetched, as part of its agentic browsing checks.
- **Documentation platforms and CMSs that generate one:** Mintlify (llms.txt and markdown page versions for every site it hosts), GitBook, Yoast SEO and AIOSEO (WordPress plugins), and Wix (a file for every Wix site).
- **Libraries and plugins:** a sample JavaScript implementation, VitePress and Docusaurus plugins, a Drupal recipe, a PHP library, a VS Code extension, and an MCP server that lets agents fetch and search llms.txt files.

Our own count: of the same 723 large sites, 158 served an llms.txt in July 2026 and 182 in October. 70 of the top 100 AI companies publish one; no other group we checked is above 18%. Those are LLMSRank’s figures, not the proposal’s.

Publishing a file is not the same as reading one. The proposal names these as publishers and tools, not as AI products that read other sites’ files. Which assistants and search engines have documented reading llms.txt, as of September 2026, is covered in [Does ChatGPT, Claude, Perplexity or Google read llms.txt?](https://llmsrank.dev/learn/does-ai-read-llms-txt).

## What it is not

- **Not robots.txt.** It grants or denies nothing. robots.txt tells automated tools what access is acceptable; llms.txt tells an agent what is worth reading. The proposal says it “can complement robots.txt by providing context for allowed content”.
- **Not a sitemap.** A sitemap lists every indexable page for search engines; llms.txt is a curated selection, and can link to other sites. See [llms.txt, robots.txt and sitemap.xml](https://llmsrank.dev/learn/llms-txt-robots-txt-sitemap).
- **Not in `/.well-known/`.** The proposal considered RFC 8615 well-known URIs and chose a plain filename instead, for the reason quoted below.
- **Not a finished standard.** It is a proposal: “The `llms.txt` specification is open for community input.”

> But well-known URIs exist only at the origin root, and many authors control only a path on a shared host: a GitHub Pages project site, for example, can publish files in its own directory but can never add one to the host’s `/.well-known/`. Like `index.html`, an `llms.txt` describes the path where it sits, something a single root location cannot express. And anyone who can publish content at a path can provide one.
>
> Source: [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)

## Frequently asked questions

### Is llms.txt an official standard?

No. It is a proposal by Jeremy Howard, first published in September 2024 and revised as v2 in August 2026, and its text says it is “open for community input”. It is not issued by a standards body such as the IETF or W3C.

### Does Google use llms.txt?

Not for Google Search. Google’s Search Central guide (llms.txt note added 2026-06-15, page last updated 2026-07-10) says Search, including its AI features, doesn’t use llms.txt: a file “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Separately, Chrome’s Lighthouse audits whether the file can be fetched.

### Do I need an llms.txt file?

Nothing requires one. It is useful if you want AI agents that visit your site to find your most useful pages quickly, in your own words, rather than piecing the site together from its HTML.

### Where do I put llms.txt?

At `/llms.txt` on your site root, or at any path such as `/docs/llms.txt`, where it covers the pages under that path. Where more than one file applies to a page, agents should use the most specific one.

### Does llms.txt replace robots.txt?

No. robots.txt says what automated access is acceptable; llms.txt points an agent to the content worth reading. The proposal describes them as having different purposes.

### Is the ## Optional section still required or meaningful?

It was never required. Since v2 it is a convention for secondary links an agent can skip, with no mechanical meaning: “they no longer carry mechanical semantics.”

### What is the minimum valid llms.txt?

An H1 with the name of the site or project, which the proposal calls “the only required section”. The summary blockquote, body text and link lists are all optional, though a file with only a title gives an agent little to go on.

### Why isn’t llms.txt under /.well-known/?

Well-known URIs exist only at the origin root, and many authors control only a path, such as a GitHub Pages project site. An llms.txt describes the path where it sits, which a single root location cannot express.

## Sources

- [llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)](https://llmstxt.org/)
- [llmstxt.org — The /llms.txt file, v2, markdown version](https://llmstxt.org/index.md)
- [llmstxt.org — Changes: v2 (August 2026)](https://llmstxt.org/changes.html)
- [Chrome for Developers — Lighthouse: llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)
- [Chrome for Developers — Lighthouse: Agentic Browsing scoring](https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring)
- [RFC 8615 — Well-Known Uniform Resource Identifiers](https://datatracker.ietf.org/doc/html/rfc8615)
- [Google Search Central — Optimizing your website for generative AI features on Google Search (last updated 2026-07-10)](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)

## Related

- [Writing an llms.txt file](https://llmsrank.dev/learn/writing-llms-txt.md)
- [Markdown page versions and link relations](https://llmsrank.dev/learn/markdown-pages.md)
- [llms.txt, robots.txt and sitemap.xml](https://llmsrank.dev/learn/llms-txt-robots-txt-sitemap.md)
- [Does ChatGPT, Claude, Perplexity or Google read llms.txt?](https://llmsrank.dev/learn/does-ai-read-llms-txt.md)