The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid.

SocioLens project,

Claim LD06-SL-C01 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

How faithfully do indices and policy documents relay AI evaluation results?The AI Index 2026 relayed counts of invalid items among 50 flagged per benchmark as invalid-question rates of 2% to 42%.

Provenance and status

Claim type: Fact
Publication status: Eligible for publication, with qualifiers: source position
Verification state: Checked against the primary source. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A claim about how the figure came to be misreported, or that other figures in the index are wrong
Limitations: The authors’ own blog post about the paper could not be read, so whether the percentage framing started there is unknown; one search found no erratum as of 15 October 2026; the paper is a preprint (its PDF footer names the NeurIPS 2025 Datasets and Benchmarks Track)
Source (as cited): Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025; Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026
Source type: Preprint, Independent research report (recorded as “Preprint + annual index report”)
Institutional position: Independent academic (Stanford); the index is also Stanford-hosted, and two of the paper’s authors are listed among its contributors
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025 (Preprint)
Read the original
Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026 (Independent research report)
No checked link to the original yet.

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): FACT (as printed: “FACT”)
Publication status (coded): ELIGIBLE (SOURCE_POSITION) (as printed: “ELIGIBLE WITH SOURCE-POSITION QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY
Source type (coded): PREPRINT, INDEPENDENT_RESEARCH_REPORT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C01	—
2	PUBLIC-FACING CLAIM	The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid.	—
3	CLAIM TYPE	FACT	base_types: FACT
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025; Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026	source_ids: SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842, SRC-SL-AI-INDEX-2026-ANNUAL-REPORT
7	SOURCE TYPE	Preprint + annual index report	classes: PREPRINT, INDEPENDENT_RESEARCH_REPORT
8	INSTITUTIONAL POSITION	Independent academic (Stanford); the index is also Stanford-hosted, and two of the paper’s authors are listed among its contributors	—
9	SCOPE	Nine benchmarks reviewed by experts in the paper; one chart and two passages in the index	—
10	DESIGN	Comparison of the index’s text and chart with the cited paper’s figure, caption and text	—
11	SAMPLE / DATASET	Per benchmark, the 50 items a statistical screen ranked most suspect, reviewed by experts	—
12	MODEL / TOOL / VERSION	40 to 91 language models used as test takers in the paper	—
13	COMPARATOR	The paper’s own figure caption and its benchmark-wide estimate	—
14	MAIN RESULT	Index: “error rates ranging from 2% on MMLU Math to 42% on GSM8K”, chart axis “Precision@50”; paper: 42 of 50 flagged GSM8K items invalid (precision 84%), 2 of 50 for MMLU Math (4%); GSM8K overall about 5% invalid (citing Vendrow et al.); the paper’s own GSM8K-Platinum labelling marks 88 of 997 (8.8%); the index text beside the chart also says “up to 84% precision”	—
15	INFERENTIAL TARGET	How faithfully an evaluation index relays a primary audit	—
16	DOES NOT LICENSE	A claim about how the figure came to be misreported, or that other figures in the index are wrong	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	The authors’ own blog post about the paper could not be read, so whether the percentage framing started there is unknown; one search found no erratum as of 15 October 2026; the paper is a preprint (its PDF footer names the NeurIPS 2025 Datasets and Benchmarks Track)	—
19	CONTRADICTORY / SUPPORTING SOURCES	Benchmark-wide error estimates of a few percent: LD06-SL-C03, LD06-SL-C04	—
20	GEOGRAPHY / LANGUAGE	Global / English	—
21	DISTRIBUTIONAL LEDGER	ThaiExam, one of the nine benchmarks, showed cultural biases and linguistic ambiguities	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	A correction by the index, or evidence that the source itself presented the counts as benchmark-wide rates	—
24	PUBLICATION STATUS	ELIGIBLE WITH SOURCE-POSITION QUALIFIER	state: ELIGIBLE · qualifiers: SOURCE_POSITION
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025” → Truong et al. (2025), “Fantastic Bugs and Where to Find…” (SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842--v1.0) — Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025; class PREPRINT (explicit public provenance pin)
“Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026” → Sajadieh et al. (2026), “The AI Index 2026 Annual Report” (SRC-SL-AI-INDEX-2026-ANNUAL-REPORT--v1.0) — Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026; class INDEPENDENT_RESEARCH_REPORT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid [Claim LD06-SL-C01, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c01/v1.0/

@misc{sociolens_ld06_sl_c01_v1_0,
  author = {{SocioLens project}},
  title = {The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2\% on MMLU Math to 42\% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2\% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5\%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8\%) invalid.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c01/v1.0/}},
  note = {Claim LD06-SL-C01, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 00f6cf2654aeb8b433310f0f796697e8e590e7f955fa15652be36af67da59636.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C01 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 01 — LD06-SL-C01” SHA-256 8bc0db37ec55d23f3b86a964133747e54b91e73f675e37098f1d572dadaaa9ac; fixture fixtures/authored/claims/LD06-SL-C01--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C01 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C01 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842--v1.0 — basis: field 6 of ENTRY 01 — LD06-SL-C01 cites "Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025"; the first field-6 segment of this ClaimVersion's repository-authored ledger block.
FIELD_6_SOURCE_REFERENCE = SRC-SL-AI-INDEX-2026-ANNUAL-REPORT--v1.0 — basis: field 6 of ENTRY 01 — LD06-SL-C01 cites "Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026"; the second field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025".
SOURCE_PROVENANCE_PIN = SRC-SL-AI-INDEX-2026-ANNUAL-REPORT--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026".