Skip to content
LLMSRank
Docs menu

Encoding and parsing

UTF-8 and parses as markdown

In short: Save the file as UTF-8 plain markdown that starts with a # title and lists links as - [name](url); the parser needs one of the two for full marks.

  • Category: Retrievability & Access · weight 15 of 100 within the category
  • Retrievability & Access is worth 15 in every profile (points of 100, by scoring profile)
  • Audit ID R4 (as shown on reports) · rubric v2.1

What it checks

Whether the file is valid UTF-8 and parses cleanly as markdown.

Why it matters

Agents read the file as markdown. Encoding errors and broken markdown turn headings and links into noise the model has to guess at.

When it applies

Always, for any llms.txt that exists.

An audit that doesn't apply, or that we couldn't verify, is removed from your score and its weight shared among the Retrievability & Access audits that do apply. It is never scored as a zero.

How it's scored

1.0 when the parser finds an H1 (# Name) or at least one markdown link list item (- [name](url)) · 0.5 when it finds neither.

How to fix

  1. Save the file as UTF-8 (a BOM is allowed but not needed).
  2. Use plain markdown: # for the title, ## for sections, - [name](url) for links.
  3. Remove HTML tags, templating leftovers and unclosed brackets.

Frequently asked questions

Is a byte-order mark (BOM) at the start of llms.txt allowed?
Yes. llms.txt v2 lists an optional BOM as the first element of the file, and we strip it before parsing, so it never hides your H1. It isn’t needed for UTF-8, so leaving it out is simplest.

Sources: llms.txt (v2, Aug 2026) · RFC 3629 §6: Byte order mark

What does the Encoding and parsing audit actually check today?
We decode the file as UTF-8 and parse it as markdown. Today it scores 1.0 when the parser finds an H1 (# Name) or at least one markdown link list item (- [name](url)), and 0.5 when it finds neither; the rubric’s per-error scaling isn’t applied yet.

Sources: Encoding and parsing · LLMSRank method

My file was saved as Windows-1252 or Latin-1. Does that matter?
Yes, for readers: we and most agents decode the bytes as UTF-8, and non-ASCII characters in another encoding (curly quotes, accented names, €) turn into replacement characters. Re-save the file as UTF-8 and declare charset=utf-8 in the Content-Type.

Sources: RFC 3629: UTF-8 · Encoding and parsing

Can I start llms.txt with an HTML comment such as a version stamp?
Yes. Leading <!-- … --> comments are skipped before we look for markup or parse the file, so <!-- last-updated: 2026-09-01 --> above your H1 is fine. Any other tag at the top, such as <html> or <div>, makes the response count as a web page, and it is not scored as an llms.txt.

Sources: llms.txt (v2, Aug 2026) · Encoding and parsing

In your research, how many llms.txt files parse cleanly?
Of the 156 tracked sites, 98.7% (154) earn full marks, with a mean score of 0.994. The two that don’t have a file where the parser finds neither an H1 nor a single markdown link.

Sources: LLMSRank method · Encoding and parsing