About llms.txt
llms.txt is a proposed markdown file that a website publishes to give AI agents a short, curated guide to its content: a title, a summary, and lists of links to pages worth reading, ideally in markdown. It was proposed by Jeremy Howard in September 2024 and revised as v2 in August 2026. This page explains the proposal as its own text does, quotes it where the wording matters, and says plainly where it is silent.
What it is
We propose adding a `/llms.txt` markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files.
It is one text file, written in markdown so that a language model can read it directly, but with a fixed structure so that ordinary code can parse it too:
llms.txt markdown is human and LLM readable, but is also in a precise format allowing fixed processing methods (i.e. classical programming techniques such as parsers and regex).
The proposal pairs the file with a second idea: pages that agents might need should have a clean markdown version at a predictable URL. That convention, and how pages advertise it, has its own page.
Why it exists
The proposal’s background starts from how agents already use the web: a coding agent fetching a library’s documentation, a chat assistant with search reading a product’s pages. Its argument is that web pages are an expensive way to feed them:
But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money. Agents are best served by concise, expert-level information gathered in a single, accessible location.
Software documentation is where the file is used most, but the proposal names other fits too:
llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials. The same structure works anywhere agents need a guided path into a site’s content: a business outlining its structure and policies, a personal site answering questions about someone’s CV, or a school providing access to course information.
Where it lives
The llms.txt file spec is for files named `llms.txt`, at the root path `/llms.txt` of a website or at any subpath (e.g. `/docs/llms.txt`). A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one.
- The name is fixed:
llms.txt. - The place is flexible: the site root (
https://example.com/llms.txt) or any path (https://example.com/docs/llms.txt). - A file covers everything under its path.
/docs/llms.txtcovers every page in/docs/;/llms.txtcovers the whole site. - The most specific file wins. A page at
/docs/api/authcovered by both/llms.txtand/docs/llms.txtshould be read with/docs/llms.txt.
Subpath files are what let people who control only part of a host take part: a GitHub Pages project site, say, or a docs section run by a different team. The proposal doesn’t say anything about subdomains. A file at docs.example.com/llms.txt covers paths on that host; nothing in the text makes it cover example.com.
The format
A file following the proposal contains these elements, as markdown, in this order:
| # | Element | Required? | What the proposal says |
|---|---|---|---|
| 1 | Byte-order mark (BOM) | No | “An optional byte-order mark (BOM)” |
| 2 | H1: # Name | Yes, the only required element | “An H1 with the name of the project or site. This is the only required section” |
| 3 | Blockquote: > Summary | No | “A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file” |
| 4 | Body: paragraphs, lists | No | “Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information about the project and how to interpret the provided files” |
| 5 | H2 sections of file lists | No | “Zero or more markdown sections delimited by H2 headers, containing “file lists” of URLs where further detail is available” |
Each entry in a file list is a markdown list item with a required link and optional notes after a colon:
Each “file list” is a markdown list, containing a required markdown hyperlink `[name](url)`, then optionally a `:` and notes about the file.
The proposal’s own mock example:
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)H2 section names are free text: ## Docs, ## Services, ## Policies. One name has a convention attached:
The “Optional” section is used, by convention, for secondary information: links an agent can skip when a shorter context is needed.
In v1, ## Optional had a special meaning for tools that expanded a file into one big context. v2 removed those tools from the proposal, and the special meaning with them:
Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics.
The proposal doesn’t say which Content-Type to serve the file with, how large it may be beyond “small enough to fit in context”, whether links must be absolute, or whether to date the file. Those are choices, and where LLMSRank’s rubric has an opinion on them it is marked as ours on the writing guide.
How agents use it
v1 made no recommendation about how the file should be processed. v2 states the expectation:
Agents are expected to view or search `llms.txt` to find the information they need, then follow the relevant links. The links in an llms.txt file should therefore point to LLM-friendly content, such as the markdown versions of pages described above. The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed.
Two consequences for authors: the file is an index, not the content, so keep it short; and each link should lead somewhere an agent can read cheaply, which usually means a markdown version of the page.
The proposal also says the file is read on demand rather than harvested for training: “Our expectation was that llms.txt would mainly be useful for *inference* rather than *training*, and that is how it has been used, though training runs could take advantage of the information too.”
Who publishes it
The v2 background describes “thousands of sites” publishing a file, and names these:
- AI labs, for their own developer docs: OpenAI, Anthropic and Gemini.
- Chrome’s Lighthouse, which audits whether a site’s file can be fetched, as part of its agentic browsing checks.
- Documentation platforms and CMSs that generate one: Mintlify (llms.txt and markdown page versions for every site it hosts), GitBook, Yoast SEO and AIOSEO (WordPress plugins), and Wix (a file for every Wix site).
- Libraries and plugins: a sample JavaScript implementation, VitePress and Docusaurus plugins, a Drupal recipe, a PHP library, a VS Code extension, and an MCP server that lets agents fetch and search llms.txt files.
Our own count: of the same 723 large sites, 158 served an llms.txt in July 2026 and 182 in October. 70 of the top 100 AI companies publish one; no other group we checked is above 18%. Those are LLMSRank’s figures, not the proposal’s.
Publishing a file is not the same as reading one. The proposal names these as publishers and tools, not as AI products that read other sites’ files. Which assistants and search engines have documented reading llms.txt, as of September 2026, is covered in Does ChatGPT, Claude, Perplexity or Google read llms.txt?.
What it is not
- Not robots.txt. It grants or denies nothing. robots.txt tells automated tools what access is acceptable; llms.txt tells an agent what is worth reading. The proposal says it “can complement robots.txt by providing context for allowed content”.
- Not a sitemap. A sitemap lists every indexable page for search engines; llms.txt is a curated selection, and can link to other sites. See llms.txt, robots.txt and sitemap.xml.
- Not in `/.well-known/`. The proposal considered RFC 8615 well-known URIs and chose a plain filename instead, for the reason quoted below.
- Not a finished standard. It is a proposal: “The
llms.txtspecification is open for community input.”
But well-known URIs exist only at the origin root, and many authors control only a path on a shared host: a GitHub Pages project site, for example, can publish files in its own directory but can never add one to the host’s `/.well-known/`. Like `index.html`, an `llms.txt` describes the path where it sits, something a single root location cannot express. And anyone who can publish content at a path can provide one.
Frequently asked questions
Is llms.txt an official standard?
Does Google use llms.txt?
Do I need an llms.txt file?
Where do I put llms.txt?
/llms.txt on your site root, or at any path such as /docs/llms.txt, where it covers the pages under that path. Where more than one file applies to a page, agents should use the most specific one.Does llms.txt replace robots.txt?
Is the ## Optional section still required or meaningful?
What is the minimum valid llms.txt?
Why isn’t llms.txt under /.well-known/?
Sources
- llmstxt.org — The /llms.txt file, v2 (modified 2026-08-10)
- llmstxt.org — The /llms.txt file, v2, markdown version
- llmstxt.org — Changes: v2 (August 2026)
- Chrome for Developers — Lighthouse: llms.txt audit
- Chrome for Developers — Lighthouse: Agentic Browsing scoring
- RFC 8615 — Well-Known Uniform Resource Identifiers
- Google Search Central — Optimizing your website for generative AI features on Google Search (last updated 2026-07-10)