On GSM1k, a newly written set of grade-school maths problems built to mirror GSM8K, some model families showed accuracy drops of up to 8% against GSM8K, consistent with partial memorization, but many models, especially frontier ones, showed minimal signs of overfitting.

SocioLens project,

Claim LD06-SL-C07 (v1.0)

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: institutional position
Verification state: Checked against the abstract of the primary source only. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A general contamination rate; conclusions for other benchmarks
Limitations: Read in abstract only; one task family; the authors’ company sells evaluation data
Source (as cited): Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332
Source type: Peer-reviewed conference paper (recorded as “Peer-reviewed conference paper”)
Institutional position: Commercial data and evaluation vendor (Scale AI)
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332 (Peer-reviewed conference paper)
Read the original

Version

This page shows version v1.0. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.

This pinned version is also the current version.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (INSTITUTIONAL_POSITION) (as printed: “ELIGIBLE WITH INSTITUTIONAL-POSITION QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (ABSTRACT_ONLY)
Source type (coded): PEER_REVIEWED_CONFERENCE_PAPER
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C07	—
2	PUBLIC-FACING CLAIM	On GSM1k, a newly written set of grade-school maths problems built to mirror GSM8K, some model families showed accuracy drops of up to 8% against GSM8K, consistent with partial memorization, but many models, especially frontier ones, showed minimal signs of overfitting.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332	source_ids: SRC-SL-ZHANG-ET-AL-GSM1K-ARXIV-2405-00332
7	SOURCE TYPE	Peer-reviewed conference paper	classes: PEER_REVIEWED_CONFERENCE_PAPER
8	INSTITUTIONAL POSITION	Commercial data and evaluation vendor (Scale AI)	—
9	SCOPE	Elementary mathematical reasoning	—
10	DESIGN	New held-out benchmark matched to GSM8K on difficulty and form	—
11	SAMPLE / DATASET	GSM1k problems (the count is not given in the abstract)	—
12	MODEL / TOOL / VERSION	Leading open and closed models as of 2024	—
13	COMPARATOR	The same models on GSM8K	—
14	MAIN RESULT	Accuracy drops of up to 8%; Spearman’s r² = 0.36 between a model’s probability of generating GSM8K items and its GSM8K–GSM1k gap; minimal overfitting for many frontier models	—
15	INFERENTIAL TARGET	Contamination as a cause of inflated benchmark scores	—
16	DOES NOT LICENSE	A general contamination rate; conclusions for other benchmarks	—
17	VERIFICATION STATE	CONTENT VERIFIED ABSTRACT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: ABSTRACT_ONLY · verified_by: UNKNOWN
18	LIMITATIONS	Read in abstract only; one task family; the authors’ company sells evaluation data	—
19	CONTRADICTORY / SUPPORTING SOURCES	Counterweight to blanket contamination claims; compare LD06-SL-C03 on label errors	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Held-out replications for other benchmarks and newer models	—
24	PUBLICATION STATUS	ELIGIBLE WITH INSTITUTIONAL-POSITION QUALIFIER	state: ELIGIBLE · qualifiers: INSTITUTIONAL_POSITION
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332” → Zhang et al. (2024), “A Careful Examination of Large…” (SRC-SL-ZHANG-ET-AL-GSM1K-ARXIV-2405-00332--v1.0) — Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). On GSM1k, a newly written set of grade-school maths problems built to mirror GSM8K, some model families showed accuracy drops of up to 8% against GSM8K, consistent with partial memorization, but many models, especially frontier ones, showed minimal signs of overfitting [Claim LD06-SL-C07, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c07/v1.0/

@misc{sociolens_ld06_sl_c07_v1_0,
  author = {{SocioLens project}},
  title = {On GSM1k, a newly written set of grade-school maths problems built to mirror GSM8K, some model families showed accuracy drops of up to 8\% against GSM8K, consistent with partial memorization, but many models, especially frontier ones, showed minimal signs of overfitting.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c07/v1.0/}},
  note = {Claim LD06-SL-C07, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 9b061dc9288dd050d4ba1194c8d1ca5a54e3d44ff0098b76b195ec9b4e98f6e7.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C07 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 07 — LD06-SL-C07” SHA-256 81f61be4358d8288f71a9e0cbde18f46e67ea20c927d57c947721fbc1e9441a7; fixture fixtures/authored/claims/LD06-SL-C07--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C07 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C07 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-ZHANG-ET-AL-GSM1K-ARXIV-2405-00332--v1.0 — basis: field 6 of ENTRY 07 — LD06-SL-C07 cites "Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-ZHANG-ET-AL-GSM1K-ARXIV-2405-00332--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Zhang, Da, Lee, Robinson, Wu, Song et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic,” NeurIPS 2024 Datasets and Benchmarks Track, arXiv:2405.00332".