39.5% and 47.1% describe the same test. One has the wrong denominator.

What did the index measure?

The index, last updated 29 September 2026, was built by HasData, a company that sells a web-scraping API. That is worth stating up front: the vendor’s product is the thing being measured, and a finding that AI-crawler blocks mostly do not work is commercially convenient for it. The data is still open, and the open data is what this piece checks.

The sample is 10,894 registrable domains, made of 9,746 from a Tranco top-web list plus 1,148 news homepages. A smaller subset of 2,096 domains got a live enforcement test: one request carrying GPTBot’s user-agent string and one carrying a Chrome string, both from a single United States datacentre IP.

The project names its own main limit plainly. The GPTBot requests “carry the official user-agent string but are not OpenAI’s verified crawler”. A site that checks whether a request really comes from OpenAI’s published address ranges, which is the documented way to verify the crawler, would correctly refuse this test traffic and be scored on the wrong basis.

Where does 39.5% come from?

The README sentence is specific, and specific is good:

39.5% of the sites that disallow GPTBot in robots.txt served it the page anyway when it knocked (234 of 592 cleanly tested sites in July). By September the share was 34.9%.

That is a clean construction. The denominator is 592: sites that both disallow GPTBot in robots.txt and returned a usable result. 234 of them served a page. 234 divided by 592 is 39.53%.

Where does 47.1% come from?

The chart’s caption reads:

Horizontal bar chart of GPTBot enforcement responses among sites that disallow it, with OK at 47.1%, hard block 22.2%, HTTP 402 10.9%, challenge 5%, rate-limited 2.4%, CAPTCHA 2.3%, server error 0.6%, other 9.5%

“Among sites that disallow it” is the denominator the sentence used. But the raw file behind the July run reports 2,083 tested records and 982 of them classed ok. 982 divided by 2,083 is 47.14%.

Every other bar lands the same way. The arithmetic is the whole argument, so here it is in full:

Chart bar	Raw count in the file	Count / 2,083	Chart’s published value
OK	982	47.14%	47.1%
Hard block	462	22.18%	22.2%
HTTP 402	227	10.90%	10.9%
Challenge	104	4.99%	5%
Rate-limited	51	2.45%	2.4%
CAPTCHA	48	2.30%	2.3%
Server error	12	0.58%	0.6%
Other (remainder)	197	9.46%	9.5%

Eight bars, eight matches, one denominator. The chart is a breakdown of what happened to every site in the enforcement run. It is not a breakdown of what happened at the sites that had asked GPTBot to stay out.

What do the 423 missing measurements do to the number?

There is a second denominator problem inside the same file, and it cuts the other way.

Of those 2,083 tested records, 423 carry no HTTP status at all. The file lists them as proxy failures: 317 ProxyError, 98 ReadTimeout, 8 SSLError. Its own status table confirms it, recording null 423 times. Those are not sites that allowed or refused anything. They are requests that never arrived.

The same file reports excluded_proxy_errors: 19. So 19 proxy failures were taken out of the 2,102 total records, and 423 were left in.

Drop them and the share moves again:

Denominator	What it contains	Share served a page
592	Sites that disallow GPTBot and returned a usable result	39.5%
2,083	Every tested record, proxy failures included	47.1%
1,660	Every tested record with an actual HTTP status	59.2%

Three defensible numbers, one test. The index’s own July-versus-September table reports 59.2% on its all-domains row, so the third figure is the project’s own rather than an outside reinterpretation. That is a useful signal: the published spread is not really a mistake, it is three different questions sharing one headline.

Does blocking GPTBot keep you out of Google’s AI answers?

The index’s most quoted claim is about citations, and it is where the sample gets thinnest:

27 of the 52 (51.9%) disallow at least one AI crawler in robots.txt. Across the sample as a whole, the same rate is 15%. Sites that block AI crawlers get cited by Google AI Mode more than three times as often as the sample baseline.

That rests on ten queries, run once in July and, by the page’s own note, not repeated in September. Ten queries returned 82 unique cited domains, 52 of which were in the sample. So the 51.9% is 27 domains.

There is also a surface question. The script that collected it is named ai_overview_citations.py, its docstring is headed “Test 4: AI Overview / AI Mode citations vs GPTBot blocking”, and the endpoint it calls is https://api.hasdata.com/scrape/google/ai-mode. AI Overviews and AI Mode are different surfaces with different triggering. The code queried one of them; the file name and parts of the write-up name both.

The direction of the finding is still plausible, and the mechanism the page offers for it is sound: a site can refuse OpenAI’s crawler and Google will keep grounding answers in its own search index, which it built with Googlebot. But “more than three times the baseline” is a ratio between 51.9% and 15% computed on 52 domains from ten news-shaped queries. It is a hypothesis with a sample attached, not a rate.

What do Google’s own pages say these controls do?

The index’s FAQ makes one claim that Google’s documentation does not support:

Google-Extended is Google’s directive controlling whether pages already in Google’s search index can be used to ground AI Overview and AI Mode answers.

Google’s crawler documentation, last updated 14 July 2026, describes Google-Extended differently. In Google’s words it manages whether crawled content “may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI”. The products Google names there are Gemini Apps and Vertex AI. The same entry adds that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”.

For the Search surfaces, Google points somewhere else. Its Search generative AI control help page says that control governs inclusion in AI Overviews, AI Mode and generative AI features in Google Discover, that it lives under Settings in Search Console, and that as of 31 August 2026 it has been rolled out to all websites worldwide. Include is the default for every property. The page also states the control “doesn’t affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended”.

So Google splits the job across two controls, and the index attributes both halves to one of them. If you blocked Google-Extended in order to stay out of AI Overviews, Google’s own page says that is not what you changed.

What should you take from the index?

The robots.txt collection is the strongest part of the work, and the publisher-versus-open-web gap is the part least sensitive to the denominator question: by the README’s figures, news publishers disallow GPTBot at 50.5% against 7.9% on the open web. The project also recorded something specific and dated after Cloudflare’s 15 September changes, finding that managed block lines had gone from 118 robots.txt files it held archived copies for.

What does not survive is the single-number version. “Around 40% of AI blocks are unenforced” and “around 47% of AI blocks are unenforced” are both in the primary source, a line apart, and the raw file says why. When an index publishes its counts, read the counts.

Methodology limits of this piece

Every figure above was read from the live pages and raw files on 10 October 2026 and recomputed. Three things it does not establish. First, whether the 227 HTTP 402 records sit inside the 462 blocked count cannot be told from the file, so the “other” row is a remainder rather than a labelled class. Second, the chart image itself was not opened; its values come from the caption text in the README. Third, none of this tests whether robots.txt blocks work against the real GPTBot, because the index did not use it.