LLMSRankBot and LLMSRank-User

LLMSRankBot

Automated research · obeys robots.txt

Mozilla/5.0 (compatible; LLMSRankBot/0.1; +https://llmsrank.dev/bot)

Used for everything that runs without a person asking: our research dataset and scheduled monitoring. For research it also sends HEAD requests to a few links listed in your llms.txt, to check that they resolve. Scheduled fetches only happen for sites whose owner has verified control (via a DNS TXT record or an llms.txt comment) and has a paid monitoring plan; to learn more, see /account. LLMSRankBot obeys robots.txt (RFC 9309) for all of it. It follows a group naming LLMSRankBot if you have one, otherwise your * group. If your robots.txt returns a server error or can’t be reached, it fetches nothing else from your site. To stop LLMSRankBot, add a robots.txt rule, or remove the verification proof (DNS TXT or llms.txt comment), which stops scheduled monitoring of your site.

To block LLMSRankBot only, add:

User-agent: LLMSRankBot
Disallow: /

LLMSRank-User

Scans a person asked for · acts like a browser tab you opened

Mozilla/5.0 (compatible; LLMSRank-User/0.1; +https://llmsrank.dev/bot)

Used only when a person starts a scan of your domain: on llmsrank.dev, or through our MCP server or WebMCP tools acting on their request. It acts on a person’s request rather than crawling, so a scan doesn’t consult robots.txt. Vendors differ on this: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, while Anthropic says its bots, including Claude-User, honour robots.txt. Once per scan it fetches the same handful of files, plus the head of up to three pages linked from your llms.txt and a check (HEAD request) for their markdown versions. The llms.txt v2 proposal fits this: it describes llms.txt as used on demand for inference rather than for training (it says nothing about robots.txt duties for fetches a person requests). To block it, match LLMSRank-User at your server, CDN or firewall.

The llms.txt draft and markdown versions tools are different: they read more of your site, so they do honour robots.txt for the LLMSRank-User token (RFC 9309; a * group applies if you have none for it). A page your robots.txt disallows for it is skipped and never fetched, and the result lists what was skipped. If robots.txt disallows the homepage, or can’t be read because of a server error, no draft is made. They still read /llms.txt and /robots.txt themselves. To stop those tools, add a User-agent: LLMSRank-User group to robots.txt.

AI crawler reachability check (advisory, not scored)

A scan also checks whether your server, CDN or firewall serves the AI crawlers that your robots.txt allows. It fetches your homepage and up to 3 key pages (from your llms.txt links, else your sitemap; same site only) once with a browser user agent, then once as each crawler listed below that your robots.txt allows on that page. It never fetches as a crawler your robots.txt disallows. Every request is a plain GET with the same headers apart from the User-Agent, and no cookies or credentials. The report compares status, redirects, page size and challenge pages, and lists crawlers that were blocked, challenged or served a different page.

Limits per scan: at most 32 requests, 256 KB read per response and 4 MB in all, 2 requests at a time with 250 ms between starts, 6 s per request and 20 s for the whole check. Anything not finished in time is reported as not measured. The exact User-Agent strings:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 LLMSRank-User/0.1 (+https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; GPTBot; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; OAI-SearchBot; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; ChatGPT-User; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; ClaudeBot; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; Claude-User; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; PerplexityBot; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)
Mozilla/5.0 (compatible; Perplexity-User; LLMSRank-User/0.1 reachability check; +https://llmsrank.dev/bot)

Each contains the crawler’s name, so rules that match on it apply, but none is the vendor’s own user agent: each says it is LLMSRank-User and links here. A firewall that verifies crawlers by their published IP addresses may treat the real crawler differently; the report says so. To block these requests, match LLMSRank-User, which all of them contain.

Other advisory checks (not scored)

Three more checks read the pages the reachability check already fetched with the browser user agent (up to 256 KB of each, held in memory for the scan and never stored): whether the content is in the HTML without JavaScript (a heuristic estimate: we run no browser), freshness (the sitemap’s lastmod dates, each page’s own dates, its Last-Modified and ETag headers) and crawl cost (time to first byte, HTML size, redirects, canonicals). So that those pages are there, the scan fetches each of them with the browser user agent even where your robots.txt allows none of the crawlers above, or can’t be read; it still never fetches as a crawler your robots.txt disallows. That browser request follows redirects one hop at a time (at most 10, each counted in the limits above) so the report can show the chain.

The crawl check also makes requests of its own, with the same browser user agent: one GET for a path that can’t exist, /llmsrank-soft-404-check- followed by a random string, to see whether a missing page answers 404 (at most 64 KB read), and one GET for each of up to 10 URLs spread across your sitemap (same site, pages already fetched skipped), reading the status and headers only. Limits: at most 24 requests counting every redirect hop (at most 5 per URL), 2 at a time with 250 ms between starts, 6 s per request and 12 s in all. All advisory checks together stop after 35 s; anything not finished is reported as not measured.

More advisory checks (not scored)

Two checks only read those same pages: structured data (the schema.org JSON-LD each kind of page calls for, and whether the facts it states also appear on the page) and answerability (question headings with a short direct answer, lists, tables, and a visible author and date).

Three more make a few requests of their own, with the same browser user agent and only where your robots.txt allows LLMSRank-User (if it can’t be read, none): one GET of each page’s Markdown version, the one it declares or else its .md URL (at most 4, 256 KB read each, 8 requests counting redirects, 10 s in all); one GET of each of /.well-known/ai-plugin.json, /.well-known/ai-catalog.json, /.well-known/ucp and /.well-known/mcp-registry-auth (16 KB read each, 8 s in all); and, when none of the pages above is your about page, one GET of the about page your homepage links to (up to 256 KB). They run one at a time with 250 ms between starts, alongside the crawl check’s requests, so at most 3 requests to your site are in flight at once. Then, for each sameAs profile link in your Organization JSON-LD that is on another site (at most 6), one HEAD request (2 at a time, 4 s each, 8 s in all) to see whether it still answers; the profile page itself is never read. All of this stays within the 35 s cap above.

Usage and licensing signals, and product data (advisory, not scored)

The usage signals check reads the robots.txt the scan already fetched (Content-Signal, Content-Usage and License lines), the response headers and meta tags of the pages above (TDMRep’s tdm-reservation and tdm-policy, and RSL licence links), and makes one GET each of /.well-known/tdmrep.json and /ai.txt with the same browser user agent, only where your robots.txt allows LLMSRank-User (32 KB read each, at most 6 requests counting redirects, 6 s in all, in the same one-at-a-time sequence as the requests above). Licence documents are never fetched. The product data check makes no requests: it reads the product pages among the pages above. Sources, as we read them on 2026-10-02: Cloudflare managed robots.txt and Content Signals Policy (dated 2026-08-03); IETF AI Preferences: attaching preferences (draft-ietf-aipref-attach-05) and vocabulary (draft-ietf-aipref-vocab-08) (dated 2026-08-19 (attach-05); 2026-09-14 (vocab-08)); W3C TDMRep, Final Community Group Report (dated 2024-05-10); Really Simple Licensing (RSL) 1.0 (dated 2025-12-10); Spawning ai.txt (Unreachable when fetched on 2026-10-02 (empty page; spawning.ai/ai-txt was under maintenance): we report whether /ai.txt exists and do not interpret it.); and for product data, Google Search Central’s merchant listing structured data page (read 2026-10-02; no last-updated date in the copy fetched 2026-10-02).

Footprint scan of your other hosts (advisory, not scored)

When the owner of a verified site starts a footprint scan (at most one per site per day), we look for the business’s other hosts only in what the site itself links to: one GET of the verified site’s homepage (up to 512 KB read), plus the llms.txt and sitemap sample its latest scan already stored. We keep subdomains of the site’s own domain and pages on known help, docs, status, careers and blog platforms that it links to, and make one DNS CNAME lookup per subdomain found, to name the platform hosting it. We do no brute-force subdomain enumeration, no certificate-transparency lookups and no DNS walking: a host the site does not link to is never found. For each host, up to 10, one at a time, as LLMSRank-User: one GET of /robots.txt (up to 256 KB), one GET of /llms.txt (up to 64 KB; not on a platform’s shared host) and the reachability check above on the host’s root page only (the same user agents, robots.txt obeyed per crawler, 10 s), at most 10 requests per host (redirects followed, at most 10 hops each), 6 s per request and 150 s for the whole scan.

Community accounts, video and audio in the footprint scan (advisory, not scored)

The same footprint scan then reads the community accounts the site’s homepage, llms.txt and homepage JSON-LD sameAs link to. It never searches a platform and never scrapes one: it calls only official APIs, and only those LLMSRank has switched on, at most 2 accounts per platform: the YouTube Data API (one channel lookup, the latest 10 uploads and their caption flags and description lengths; at most 6 quota units a scan; YouTube data is deleted after 30 days), the GitHub REST API (the owner’s public profile and its 10 most recently pushed repositories; at most 4 requests) and the Stack Exchange API (a tag’s question counts or a user’s reputation; at most 4 requests). Reddit is never called: its Data API needs Reddit’s approval and an agreement for commercial use, so Reddit accounts are only listed. Other community links (Discord, Slack, forums and the like) are listed, never visited. Then one GET, as LLMSRank-User, of each podcast feed the homepage or llms.txt points to (at most 2, 512 KB read each) for transcripts and episode descriptions. 6 s per request and 25 s for all of it. API terms as we read them on 2026-10-02. The media advisory check of a normal scan (image alt text, video and audio markup) makes no requests: it reads the pages above.

Pages AI engines cite instead (Daily Pro)

For sites with citation tracking, a weekly job looks at the third-party pages AI answers cited for the site’s own questions and checks whether each one mentions the business. Up to 15 pages per site per week (at most 2 per site cited), each not checked in the last 7 days, as LLMSRankBot: robots.txt is read first and obeyed, then one GET of the page (300 KB read, 8 s, 120 s for the whole run). Across all sites at most 3,000 such pages are fetched per week (our default cap). Only whether the page names the business is stored, never its text.

Connected accounts: listings, reviews and Bing (weekly)

Only for verified, monitored sites whose owner connected Google or Bing Webmaster Tools and mapped a property to the site; without a connection nothing here runs. Once a week, through the owner’s own connection: for a mapped Google Business Profile location, one read of the location (name, address, phone, hours, category, website) and one read of its reviews list (5 reviews, from which we keep only Google’s review count, average rating and the date of the newest review; review text and reviewers’ names are never stored, and these numbers are deleted after 30 days); for a mapped Bing site, three reads (crawl stats, crawl issues and sitemaps). These are API calls to Google and Bing, not requests to your site. To compare the listing with your site, we then fetch your homepage and up to 2 contact-like pages it links to, as LLMSRankBot: robots.txt is read first and obeyed (a page it disallows, or every page when it can’t be read, is not fetched), 512 KB read per page, 8 s per request. For the Bing comparison, one GET of your /sitemap.xml (and up to 6 same-host child sitemaps of a sitemap index, up to 5 MB each, at most 50,000 URLs counted), the same way.

IndexNow key check and submissions (owner opt-in)

Only for monitored sites whose owner saved their own IndexNow key; we never create or host a key. After each monitoring rescan of the site, one GET of the key file the owner publishes at https://<your domain>/<key>.txt as LLMSRankBot, robots.txt obeyed (up to 512 bytes read, 10 s). If the file is in place and the owner opted in to submissions, we read your /sitemap.xml as above and send the URLs on your verified host that are new or whose lastmod changed, in one POST to https://api.indexnow.org/indexnow, IndexNow’s shared endpoint, which passes them on to the search engines that take part. The first run only records a baseline and sends nothing. Limits: at most 1,000 URLs per submission, one submission per site every 6 hours and 3 per site per day, and 60 per hour across all sites. Every attempt is logged for the owner.

Reviews (advisory, not scored)

We never fetch a review site’s pages (G2, Capterra, Trustpilot, Yelp, Tripadvisor, Healthgrades and the like) and never search one. The footprint scan lists the review profiles your homepage, llms.txt and JSON-LD sameAs link to, from the homepage it already fetched: no extra request. Google review numbers come only from the owner’s connected Business Profile, as described above. The review markup check of a normal scan makes no requests: it reads the pages above.