Skip to content
LLMSRank
Learn menu

How to check your llms.txt

Scan your domain for free on LLMSRank to get a graded report, or check by hand: fetch the file, confirm robots.txt lets agents reach it, then check its structure, links and size against the steps below. No check, ours included, can tell you whether a particular AI engine reads the file.

Updated

The quick way: a free scan

Scan a domain or a page URL and LLMSRank fetches the site’s robots.txt, llms.txt, llms-full.txt, AGENTS.md and sitemap.xml, finds the most specific llms.txt that covers the URL, and scores it against every audit that applies to the site. The report leads with anything that stops agents fetching the file, then the grade and the fixes worth the most points. It is free, up to 5 scans a day, with no account.

Chrome’s Lighthouse also has an llms.txt audit, in its experimental Agentic Browsing category. It checks that the file can be fetched, failing on a server error and marking a missing file as not applicable; it doesn’t read what the file says.

The manual checklist

Each step names the LLMSRank audit that checks it; the audit page says how it is scored and how to fix it. The proposal itself requires only the H1. The rest is what the proposal describes or what our rubric recommends, and each audit page says which.

1. Fetch it. Request the file the way an agent would, and look at the status line and headers:

curl -sI https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -40
  • A file covers the page you care about, at the root or at its path, such as /docs/llms.txt: Covering llms.txt.
  • It answers 200 with no login and at most one same-host redirect: Clean fetch.
  • It is served as text/plain or text/markdown, not an HTML page: Content type.
  • It is valid UTF-8 and parses as markdown: Encoding and parsing.

2. Check robots.txt. Open /robots.txt and read the groups that apply to the AI crawlers you want, plus User-agent: * for any that have no group of their own.

  • No group you care about disallows /llms.txt; if one disallows everything, add Allow: /llms.txt to it: AI crawler access.
  • The AI crawlers you want are named explicitly, not covered only by User-agent: *:Named AI crawlers.

3. Check the structure.

4. Check the links.

5. Check size and dates.

  • The file fits in about 8,000 tokens, roughly 32,000 bytes: Token budget.
  • If your content changes, the file says when it was last updated: Freshness signals.

Checking from an AI agent

An agent can run the same checks without the scan form.

  • In a browser with WebMCP, every llmsrank.dev page registers tools on the page. check_llms_txt takes the text of an llms.txt, such as a draft you are writing, and reports its structure: H1, summary, sections, link format and descriptions, links to markdown, URL hygiene and size. It fetches nothing. start_scan runs a real scan, counted against the visitor’s free scans, and get_report returns the finished report in markdown.
  • Without WebMCP, the same actions are plain HTTP, described in OpenAPI 3.1 at /openapi.json: POST /api/scan starts a scan, /scan/<id>.json reports progress and /scan/<id>.md is the finished report.
  • Instructions for agents are in our own /llms.txt and /AGENTS.md.

What a check can’t tell you

A checker can tell you whether agents can fetch the file and whether it is well made. It can’t tell you whether a given AI product reads it, or whether it will get you cited. As of 2026-09-29 only Google documents a position, and it is that Google Search ignores llms.txt; the other major vendors have documented nothing either way. The details, with dates, are on Does ChatGPT, Claude, Perplexity or Google read llms.txt?.

Your own server logs are the closest thing to an answer for your site: they show which user agents request /llms.txt. User-agent strings are self-declared, and a request shows a fetch, not that the file shaped an answer.

Frequently asked questions

Is there a free llms.txt checker?
Yes. LLMSRank scans any domain or page URL for free, up to 5 scans a day with no account, and returns a graded report with the fixes worth the most points.
How do I test my llms.txt by hand?
Fetch it with curl -sI and check for a 200 and a text content type, check that robots.txt doesn’t block it, then check for one H1, a summary blockquote, H2 sections and described links. Each step on this page links to the audit that explains it.
Can a checker tell me whether ChatGPT or Google reads my llms.txt?
No. A check tells you the file is reachable and well made. Google documents that Search ignores llms.txt, and OpenAI and other vendors have documented nothing either way, as of 2026-09-29.
How is this different from Lighthouse’s llms.txt audit?
Lighthouse checks that the file can be fetched: it fails on a server error and marks a missing file as not applicable. LLMSRank also reads the file and grades its contents, along with robots.txt, the sitemap and agent files.
Can an AI agent check my llms.txt for me?
Yes. In a browser with WebMCP, the check_llms_txt tool on llmsrank.dev checks pasted text and start_scan runs a full scan; without WebMCP, agents can use the HTTP endpoints described at /openapi.json.
What is the minimum for a valid llms.txt?
An H1 with the name of the site, which the proposal calls “the only required section”. Everything else in the checklist makes the file more useful to agents but isn’t required.

Sources

Related