Skip to content
LLMSRank
Learn menu

llms.txt generators: what generated files miss

A generator can turn your pages into a well-formed llms.txt, but it can’t make the editorial choices behind the checks that files in the llms.txt Index most often miss: which page answers which question, who the site is for, and what each link answers. Generate the file, then edit it.

Updated

The kinds of generator

Generators fall into three groups. The ones named here are examples of tools that exist, described from their own documentation as of 2026-09-29; this is not a ranking.

  • Documentation platforms generate the file for every site they host. Mintlify says its file “lists all available pages in your documentation”, takes each description from the page’s frontmatter description, and is replaced by a custom llms.txt if you add one. GitBook’s file “lists every published page with a Markdown version”. Both also publish llms-full.txt and markdown versions of pages.
  • CMS plugins and site builders. Yoast SEO can “automatically select key content” or let you choose it in manual mode. The llms.txt proposal also names AIOSEO for WordPress, and Wix, which generates a file for every Wix site.
  • Crawlers. These fetch your site and write the file, often with a language model. Firecrawl’s llms.txt Generator “leverages Firecrawl to crawl your website and extracts data using gpt-4o-mini”; Firecrawl says that standalone API is deprecated in favour of its main endpoints.

The llms.txt proposal lists more, including VitePress and Docusaurus plugins. What they share is where the content comes from: your pages, their metadata, or a model’s summary of them.

What generated files typically miss

Structure is the easy part: in the llms.txt Index of 187 real files, 91% earn full marks on Single H1 and 89% on H2 sections. The checks below are the ones a generator can’t do for you, lowest share of full marks first.

CheckWhat it asksFull marks in the IndexWhy a generator can’t do it for you
Intent routingRoutes each kind of question to the page that answers it1% of 167 sitesWhich questions your users ask is knowledge about your business that no single page states, so there is nothing to copy.
Freshness signalsCarries a last-updated date that agrees with your sitemap7% of 114 sitesRegenerating keeps a file current, but an agent can’t tell that unless the file says when it was updated.
LLM-friendly linksLinks point to markdown or plain text rather than HTML12% of 174 sitesA generator can only link a markdown version that exists. Some documentation platforms publish them; most sites in the Index don’t have them yet.
AI crawler accessrobots.txt lets the AI crawlers you want fetch the file20% of 183 sitesrobots.txt sits outside the file, so a well-made generated file can still be blocked.
Role and scopeSays what the site is authoritative about, for whom, and what it doesn’t cover22% of 187 sitesRole, audience and scope are statements you make; page content rarely states them in a form an agent can quote.
Link descriptionsEvery link has a description saying what the page answers29% of 174 sitesDescriptions taken from page metadata or a model’s summary say what a page is, not necessarily which question it answers.
Token budgetFits an agent’s context budget, about 8,000 tokens80% of 187 sitesListing every page of a large site can push the file past the budget.
Curation, not a sitemapSelects your best pages rather than mirroring the sitemap88% of 99 sitesA tool that lists every published page mirrors the sitemap by design; the choosing is left to you.

Good examples

Sites that earn full marks in the Index on the checks that take the most judgement. Read their files for patterns worth copying:

After you generate: what to edit

  1. Find out how to override the generated file. Mintlify, for example, documents that a custom llms.txt at the root of your project replaces the generated one.
  2. Rewrite each link description to say what the page answers, not what it is.
  3. Cut pages an agent doesn’t need, such as tag archives, pagination and near-duplicates, and put the pages you most want quoted first.
  4. Add a summary blockquote under the H1 and a short paragraph on what the site covers, for whom, and what it doesn’t.
  5. Add a section that routes common questions to the page that answers each one.
  6. Add a last-updated date, and change it when the file changes.
  7. Check robots.txt lets the AI crawlers you want fetch /llms.txt, then check the file.
  8. Paste the draft into the llms.txt editor to see, as you edit, which audits it meets from the file alone.

The writing guide has two complete example files to work from.

Frequently asked questions

What is the best llms.txt generator?
We don’t rank generators. Use the one your platform already provides, such as your documentation platform or CMS plugin, then edit its output: the checks files most often miss are editorial, and no generator makes those choices for you.
Are generated llms.txt files bad?
No. A generator is a quick way to a well-formed file. What it can’t supply is judgement: which pages matter, what each one answers, and who the site is for.
What do llms.txt files most often miss?
In the llms.txt Index, the lowest shares of full marks among these checks are Intent routing (1%), Freshness signals (7%), LLM-friendly links (12%). Each audit page explains the fix.
Should I let an AI write my llms.txt link descriptions?
A model can draft them, but check each one says which question the page answers, in your words. A description that only restates the page title gives an agent nothing to choose by.
Can I replace a generated llms.txt with my own?
Usually. Mintlify, for example, documents that a custom llms.txt at the root of your project overrides the generated one. Check your platform’s documentation.
Does a generated llms.txt help my site rank in AI search?
No vendor documents that any llms.txt does. Google documents that Search ignores the file. See Does ChatGPT, Claude, Perplexity or Google read llms.txt?.

Sources

Related