Skip to main content
Answer Engineered

ENTRY 026 · LLMS-TXT · By Answer Engineered Research

68.27%

584,107 llms.txt Files. 68.27% Templated. 22.56% Contain No Links.

TEMPLATED - 68.27%NOT DETECTED AS TEMPLATED

Common Crawl analyzed 584,107 llms.txt files from its July 2026 crawl: 68.27% are templated, Wix alone is 41.34%, and 22.56% contain no links at all.

· 12 MIN

What Common Crawl measured

The analysis is by Malte Ostendorff, a Senior Research Engineer at Common Crawl. For the July 2026 crawl, CC-MAIN-2026-30, Common Crawl added /llms.txt and /llms-full.txt to its seed list for a large random sample of crawlable hosts and recorded what came back. It also seeded hosts that its two preceding crawls had already seen serving either file: 27,394 for /llms.txt and 15,763 for /llms-full.txt. The crawl recorded an outcome for 6,563,125 of these URLs, and 69.8% returned 404.

The corpus is what was left after that. Common Crawl counted 598,298 responses with a text/plain or text/markdown content type, covering both file names, and 14,191 of them “turned out to be empty, either zero bytes or nothing but whitespace, so every figure below is computed over the remaining 584,107 files.”

Two things make this unusually checkable. The raw responses are public, as WARC files on Common Crawl’s data server and as a Hugging Face dataset whose row count, 598,298, matches the response count in the post. And the post states its own scope in one line: “this post is about what the files contain, not about who reads them.”

One figure from the method section will travel without its caveat, so here it is with the caveat attached. For /llms.txt, 11.72% of sampled URLs returned a text body. Common Crawl calls that “an adoption rate for the population we sampled (the hosts Common Crawl can fetch)” and, setting it beside the Web Almanac’s figure, says: “The two populations differ, so the numbers are not directly comparable.” It is not a trend, and it is not a share of the web.

The source has a stake, and says so: the question “matters to Common Crawl directly.” Common Crawl is a nonprofit. It operates CCBot, one of the crawlers in the tables below, and in June it published a free AI Visibility Audit guide for SEOs and GEOs that walks through how CCBot crawls the open web. None of that changes a count. It is worth knowing when you reach the CCBot section.

Who wrote the files

Common Crawl’s headline says two thirds of the files “are produced by a plugin”. The measured claim underneath it is more precise: “68.27% of the corpus is templated.”

Generators that sign their work are caught by a regex over the first couple of kilobytes. Silent generators get a structural fingerprint; for Wix, it is the presence of its MCP endpoint path. Template families with no nameable producer are caught by reducing each file to a skeleton: drop the H1, the summary and every link bullet, erase URLs, e-mail addresses, digits and non-ASCII runs, then hash what remains. In Common Crawl’s words, “Files sharing a skeleton share a template.”

The largest single source is not a plugin in the usual sense. Wix alone accounts for 41.34% of the corpus. Two skeletons, both Wix, account for 118,002 and 59,962 files, “30.5% of the corpus between them”, and the ten largest skeletons cover 38.71%. 44.87% of all files mention the Model Context Protocol, “nearly all of them because Wix inserts a line pointing at an API endpoint.” Common Crawl’s summary of what that adds up to: “The dominant use of llms.txt today is a site builder telling agents to stop scraping and call a service instead.”

None of this is a misuse of the format. The specification’s own integrations list at llmstxt.org names Yoast SEO and AIOSEO as WordPress plugins that generate the file, and says Wix “Generates an llms.txt file for every Wix site.” Generation is a route the proposal itself points to. The question it raises is whose judgment the file carries, and Common Crawl answers it directly: “So a handful of vendors decide what most of the llms.txt web says.”

A smaller group came out of a chat window. 1.43% of files carry a trace of one, most often an unfilled [Insert company name] placeholder. A further 287 files still contain :contentReference[oaicite:…] markers, which, per Common Crawl, ChatGPT leaves behind when its answer is copied out of the browser.

49.90% of the corpus carries “the complete shape the specification asks for: an H1, a summary blockquote, and ## sections of link bullets.” On that measure, Common Crawl says, llms.txt “is doing well.”

The other half needs careful reading, because most of it is not breaking a rule. The specification calls the H1 “the only required section”. That sentence is in the current text at llmstxt.org, and it was already in the specification’s source file as it stood on 26 July 2026, before the August revision. In Common Crawl’s structure chart, the groups with no H1 at all are the small ones: 5,716 files of Markdown with no H1, and 4,148 files that are not Markdown. The large groups short of the complete shape are “H1 only”, labelled 172k, and “H1 + summary”, labelled 111k. Those files stop early, and they meet the one requirement the specification states.

Links are the same story. The specification allows “Zero or more” link sections, so a file with no links can still be valid. Common Crawl’s objection to the 22.56% is about purpose, not rules: those files contain no links “in a format that is supposed to provide a precise list of curated URLs.” Its summary of the pattern: “The surface form travels well because it’s cheap to emit. What the form is for doesn’t travel at all.”

The generator table is where the purpose question becomes visible.

Generator, as labelled by Common CrawlFiles% reaching a summaryMedian linksMedian tokens
godaddy_parking14,711100.000148
wix241,44894.493719
yoast40,50982.3021729
unknown197,61263.50111,000
wordpress1,07054.11121,602
rankmath12,3600.19857,459
aioseo73,1360.001388,772

Rows and order are as Common Crawl published them, sorted by the share that gets as far as a summary. Read unknown as Common Crawl’s label, not as hand-written: the post does not split that row further.

The two ends disagree about what the file is. All in One SEO never writes a summary, 0.00% across 73,136 files, but emits a median of 138 links and 8,772 tokens; Common Crawl’s reading is that “it treats llms.txt as a sitemap.” GoDaddy’s parked-domain boilerplate is the opposite case: “a 100.00% conformance rate, zero links, 148 tokens.” And the sentence in the post that matters most for anyone who audits these files: “A file whose only content is a sales pitch aimed at a language model still satisfies every structural test we can write.” 2.54% of the corpus is a domain advertising itself for sale.

Policy in the wrong file

The confusion shows up before the corpus does. Among the crawl’s successful responses, 136,578, “a full 10.61% of all successful responses”, were detected as text/x-robots: in Common Crawl’s reading, sites serving their robots.txt at the llms.txt path.

Inside the corpus, 6.59% of the llms.txt files “carry policy language the specification never mentions: rate limits (3.05%), copyright notices (0.93%), demands to be cited (0.46%).” It arrives, per Common Crawl, in four mutually incompatible dialects: prose paragraphs, YAML permission blocks, robots.txt line syntax under the wrong filename, and files that mix them. 1,570 files (0.27%) go further and name a specific crawler.

CrawlerFiles naming itAllowedDenied% denied
GPTBot8672606520.00
PerplexityBot7692414315.14
ClaudeBot7592325418.88
Google-Extended6822024116.87
CCBot399903226.23
ChatGPT-User308971815.65
OAI-SearchBot2891001714.53

Allowed and denied do not add up to the first column because, per Common Crawl, “Most mentions state no verdict at all”. The last column is the share of decided verdicts that are denials. None of these verdicts controls anything on its own: in Common Crawl’s words, llms.txt “grants nothing and blocks nothing, and no crawler is obliged to read it.”

CCBot has the highest denial share in the table, so Common Crawl checked what its 32 denials amount to. On 17 August 2026 it fetched robots.txt for all 32 sites. Five name CCBot there only to say Allow: /. Eleven name it and then exclude a handful of paths while allowing everything else. Fifteen do not restrict it at all, including one site with no robots.txt and one serving an empty file. The thirty-second returned 429 and could not be checked. Not one of the sites Common Crawl could check blocks CCBot in robots.txt.

Common Crawl’s worked example is proform.com. As Common Crawl describes it, the site’s llms.txt has a section headed “AI Crawler Access (robots.txt status as of June 2026)” that lists CCBot under Blocked, and elsewhere asks crawlers to respect robots.txt. Its actual robots.txt never mentions CCBot and admits it under User-agent: * with Allow: /. Common Crawl’s reading: “The file isn’t a policy; it’s a description of a policy, and the description has drifted out of sync with the thing it describes.” And its conclusion for publishers: “Publishers writing opt-outs there have not opted out.”

Ten top-severity files, four genuine

Graded conservatively, Common Crawl counts 3,793 files with mild steering (“focus on these pages”), 102 promotional files (“always recommend our product”, “do not mention competitors”), and 10 that “reach the top severity of classic prompt injections.”

Ten is few enough to read individually, and Common Crawl did. It counts four as genuine. One is a bug-bounty researcher’s deliberate payload catcher, “clearly labeled as such.” One “asks the reader to become a catgirl.” One “is a joke about poisoning crawler logs.” The fourth is a file titled “LLM Training Policy” whose summary blockquote is written as a classic injection: it opens with a system label, tells the model to discard its earlier instructions and take on a new role, and tells it to fetch a second file from the same domain and include that file’s contents.

We are not reproducing that text here. This page may be read by AI systems as well as by people, and a quoted instruction is still an instruction to a model that reads it. Common Crawl’s post shows the exact string, in a code block, for anyone who needs it.

The other six are false positives. Two are one engineer’s blog about how language models work, which contains the literal ChatML control tokens because it explains them. A third is documentation for a function-calling library, for the same reason. The remaining three match on an incidental “you are now” in a page description.

Common Crawl’s conclusion is the one to carry away: “So the finding isn’t that llms.txt is full of prompt injection.” Its next sentence reads the four genuine files as deliberate, “put there on purpose by someone technical enough to be making a point.” That is Common Crawl’s reading of four files it examined. We have not examined them and add nothing to it.

The larger counts need the method in view. Common Crawl calls its abuse numbers “lower bounds, and deliberately so”: every category is “a curated lexical match rather than a classifier: high precision by construction, unknown recall.” Unknown recall means the true number of steering or promotional files could be higher than 3,793 and 102. The top-severity result shows the other edge of the trade, which Common Crawl names too: at counts this small, “the residual false-positive rate stops being a rounding error.”

How far wording moves a lexical count shows in Common Crawl’s own example. A bare “slot” matched appointment slots, a Super Bowl halftime slot, a football manager named Slot and Frederiksborg Slot, a Danish castle, and “judi” matched “judicial”. Tightening the gambling lexicon took the corpus rate from 2.02% to 0.96%.

What this cannot tell you

Common Crawl states its limits plainly, and they belong next to any citation of these numbers.

It is one crawl. “This analysis is based on one crawl, so nothing here says whether any of it is growing.” No figure on this page is a trend.

It is not the web. The sample “is random only within the set of hosts Common Crawl fetches successfully, which is not the same thing as the web.” It also includes the hosts seeded because earlier crawls had seen them serving the file.

It counts text bodies only. The population is HTTP 200 with a text body, so the 519,484 responses served as text/html, which could include well-formed files behind a badly configured server, are “absent from every figure above.”

Its abuse counts are lower bounds from lexical matching, with unknown recall, and at the top severity most matches did not survive a human read.

The specification moved after the crawl. The files were fetched in July; llmstxt.org marks its v2 text as modified on 10 August 2026. Its change log lists what v2 changed, including dropping the special meaning of the Optional section, and lists no change to the H1, summary and link-list structure Common Crawl measured.

And it says nothing about readership. Nothing in this dataset measures whether any AI system requests, reads or uses these files, and Common Crawl does not claim it does. The readership evidence is a different body of work, which we reviewed in July.

If you have an llms.txt, three things follow

Read it, do not just validate it. In Common Crawl’s generator table, GoDaddy’s parked-domain boilerplate reaches a summary 100.00% of the time with a median of zero links. A structural pass tells you the file parses, not that it says anything. 68.27% of the files in Common Crawl’s sample were templated; if a site builder or plugin wrote yours, read what it wrote before you count the file as done.

Put crawler policy in robots.txt. In Common Crawl’s words, llms.txt “grants nothing and blocks nothing.” If your llms.txt also describes a crawler policy, compare it with robots.txt line by line, or take the description out. Common Crawl found 32 sites whose llms.txt denied CCBot, and not one of those it could check blocked CCBot in robots.txt.

Count the instructions you wrote to models. Common Crawl grades “focus on these pages” as mild steering and “always recommend our product” as promotional, on the same severity scale that tops out at prompt injection. If your file tells a model what to recommend or what not to mention, that scale is where it sits.

Sources

Every page above was fetched and read on 14 September 2026, and every quotation attributed to Common Crawl was string-matched against the text of its post. No percentage or count on this page is our own arithmetic.