Skip to main content
Answer Engineered

Verified: 2026-07-18By Answer Engineered Research

llms.txt Does Not Work: 1 Cited URL in 94,614

One cited URL in 94,614 came from an llms.txt page. Why adoption never became usage, what Google says on the record, and what moves AI citations instead of it.

Published July 18, 2026

What does the citation data show?

The sharpest measurement is Allmo’s analysis of 94,614 AI-cited URLs: one citation traceable to an llms.txt page, a rate of 0.00106%. For context, a tactic with zero effect and random noise would be expected to produce roughly this result.

An independent earlier measurement points the same way. Otterly.ai’s 90-day llms.txt experiment logged 62,100 AI bot visits to its site; approximately 0.1% touched the llms.txt file. The same writeup notes a 500-million-visit dataset in which llms.txt drew 408 hits, and that no major AI vendor has committed to reading the file. Two measurements, different methods, same conclusion: the crawlers overwhelmingly do not fetch it, and the answers overwhelmingly do not cite through it.

What does Google say on the record?

Google has stated publicly that its AI systems do not use llms.txt, as documented in Allmo’s roundup of the on-record statements. This aligns with Google’s broader published position: its AI features optimization guide states that no special files or structured data are required for AI Overviews or AI Mode citations — Google’s AI surfaces draw on the same crawl and index as search. There is no confirmed commitment from OpenAI, Anthropic, or Perplexity to consume the file on their web-search surfaces either, per the Otterly experiment writeup.

Why does adoption keep growing anyway?

Because adoption and usage are different measurements, and only adoption is visible to the people recommending the tactic. Roughly 10% of domains now publish an llms.txt file, per the same June 2026 analysis. Publishing the file is cheap, feels proactive, and produces an auditable artifact — an agency can point at it. Whether any engine consumes it is invisible without log analysis, so the feedback loop that would kill the tactic never fires. The result is a growing installed base of files that, on the best available evidence, nothing on the consumer web reads.

There is one documented set of genuine consumers: IDE coding agents such as Cursor and Claude Code, which can fetch an llms.txt when a developer points them at documentation. That is a real but narrow use case — developer tooling reading developer docs — and it does not transfer to AI search visibility for a content site.

Isn’t llms.txt just robots.txt for AI?

No — and the comparison explains the confusion. robots.txt is consumed: the major AI vendors run named crawlers that honor it, and honor it separately per agent. Anthropic split its crawling into three distinct user agents in February 2026 — ClaudeBot for training, Claude-User for live retrieval, Claude-SearchBot for its search index — each respecting robots.txt independently, mirroring OpenAI’s GPTBot, OAI-SearchBot, and ChatGPT-User split. robots.txt works because the crawler operators committed to reading it and their compliance is observable in server logs.

llms.txt inverts every part of that. No vendor has committed to it, no consumption is observable at meaningful scale, and it grants nothing robots.txt does not already control. A site’s actual AI-crawler policy lives in robots.txt and its CDN rules; llms.txt is a suggestion posted where, on the log evidence, almost nobody looks.

Is there any harm in keeping the file?

On the evidence, no. The file is inert: it costs a few kilobytes, no engine penalizes it, and the developer-tooling case gives it marginal value for documentation-heavy sites. Our review classifies it as hygiene — comparable to a humans.txt — not as a visibility lever. The harm is opportunity cost: hours and audit line-items spent on a file nothing reads, while the mechanics that measurably gate AI citation go unaddressed.

Any site can also run this check on its own data rather than taking the aggregate numbers on trust: raw server or CDN access logs will show how many requests hit /llms.txt versus how many pages GPTBot, ClaudeBot, and PerplexityBot fetched over the same window. In every disclosed measurement to date, the first number rounds to zero while the second does not — a comparison that takes minutes and settles the question for one’s own domain.

What actually moves AI citations instead?

The contrast between llms.txt and the levers with measured effects is stark.

Crawler access is the real gate. Otterly.ai’s report on 1 million+ AI citations found 73% of analyzed sites had a technical barrier — robots.txt rules, CDN or WAF blocks — preventing AI crawler access outright. A site blocking GPTBot while publishing an llms.txt has the priorities exactly inverted.

Server-rendered HTML is a prerequisite. The major AI crawlers — GPTBot, ClaudeBot, PerplexityBot — execute no JavaScript, per an analysis covering 500 million+ GPTBot fetches. Content that only exists after client-side rendering does not exist for these systems.

Answer placement measurably matters. Surfer’s two-wave study of 100,000 citation placements found 38% of AI citations are pulled from a page’s first 100 words as of June 2026 — nearly double the 20% measured a year earlier. Leading with the direct answer is a documented, growing effect.

Content structure carries measured odds. The GEO-16 framework study (1,702 citations across three engines) found the pillars most associated with citation were metadata and freshness, semantic HTML, and structured data; pages hitting at least 12 of 16 pillars with a quality score of 0.70 or higher reached a 78% cross-engine citation rate. And the Princeton GEO paper (KDD 2024) measured up to a 40% visibility boost from generative-engine-optimization techniques, with adding statistics alone worth a 25.9% citation lift.

Density, freshness, and sourcing show threshold effects. SE Ranking’s 129,000-domain analysis of ChatGPT citation factors reports that pages with 19 or more discrete statistical data points averaged 5.4 citations, pages with expert quotes averaged 4.1 versus 2.4 without, and content updated within the last 3 months averaged 6.0 citations — each a larger measured effect than llms.txt has produced in any dataset our desk has reviewed.

Every one of those levers has a disclosed dataset and a measured effect size. llms.txt has one citation in 94,614. The comparison is not close.

The bottom line

As of July 2026, llms.txt is a proposal the consumer AI-search market declined to adopt on the consumption side. Keep the file if it is already published — it costs nothing — but no honest audit can list it as an AI-visibility action while crawler access, server-side rendering, and answer-first structure remain unfixed. Adoption statistics measure enthusiasm. Citation statistics measure reality. They currently disagree by roughly five orders of magnitude.

Sources

Monthly AI Visibility Index

Get the monthly index

One email a month: who AI engines cite, who they dropped, and the verified numbers behind each shift. No spam, unsubscribe any time.