On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks.

SocioLens project,

Claim LD01-SL-C22 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

Do the failure patterns of 2023–2024 language models persist in reasoning models?Counterfactual task versions degrade performance consistently.

Provenance and status

Claim type: Estimate and Interpretation
Publication status: Eligible for publication, with a qualifier
Verification state: Checked against the primary source; the source is a preprint. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A claim that models only recite, or that they reason generally; the authors find evidence of both
Limitations: Read in arXiv version 3 (28 March 2024), which carries no venue statement; the NAACL 2024 version was not compared; 2023 models; the authors note humans may also do worse in unfamiliar conditions; one model tested is made by Anthropic, the developer of the model that drafted this debate
Source (as cited): Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024
Source type: Peer-reviewed conference paper (recorded as “Peer-reviewed conference paper”)
Institutional position: Independent academic (MIT, Boston University)
Maturity of the evidence: Emerging evidence
Recorded date: 16 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024 (Peer-reviewed conference paper)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE + INTERPRETATION (as printed: “ESTIMATE / INTERPRETATION”)
Publication status (coded): ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT)
Source type (coded): PEER_REVIEWED_CONFERENCE_PAPER
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD01-SL-C22	—
2	PUBLIC-FACING CLAIM	On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks.	—
3	CLAIM TYPE	ESTIMATE / INTERPRETATION	base_types: ESTIMATE, INTERPRETATION
4	RESEARCH STRAND	B — Evidence that could tell the accounts apart	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024	source_ids: SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477
7	SOURCE TYPE	Peer-reviewed conference paper	classes: PEER_REVIEWED_CONFERENCE_PAPER
8	INSTITUTIONAL POSITION	Independent academic (MIT, Boston University)	—
9	SCOPE	Eleven task families with default and counterfactual versions; English; 2023 models	—
10	DESIGN	Default and counterfactual task pairs, with checks that models understood the changed rules	—
11	SAMPLE / DATASET	11 tasks	—
12	MODEL / TOOL / VERSION	GPT-4, GPT-3.5, Claude (claude-v1.3) and PaLM-2	—
13	COMPARATOR	The default version of each task	—
14	MAIN RESULT	Above-random counterfactual performance on most tasks, but consistently below default performance	—
15	INFERENTIAL TARGET	How far model abilities are general rather than tied to familiar task forms	—
16	DOES NOT LICENSE	A claim that models only recite, or that they reason generally; the authors find evidence of both	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Read in arXiv version 3 (28 March 2024), which carries no venue statement; the NAACL 2024 version was not compared; 2023 models; the authors note humans may also do worse in unfamiliar conditions; one model tested is made by Anthropic, the developer of the model that drafted this debate	—
19	CONTRADICTORY / SUPPORTING SOURCES	LD01-SL-C04, LD01-SL-C21	—
20	GEOGRAPHY / LANGUAGE	English task set	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-01, T4, D3	—
23	WHAT WOULD CHANGE THIS CLAIM?	Models that perform as well on counterfactual as on default versions	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-16	—

Field 22 references resolved against Field Map v0.2: LD-01 (LIVING_DEBATE); T4 (TENSION); D3 (DOMAIN)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024” → Wu et al. (2024), “Reasoning or Reciting? Exploring the…” (SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0) — Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)

Cited by Living Debate versions

What kind of thing is generative AI, and does the answer predict anything? (v0.1) LD-01

Cite this page

SocioLens project. (2026, October 16). On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks [Claim LD01-SL-C22, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld01-sl-c22/v1.0/

@misc{sociolens_ld01_sl_c22_v1_0,
  author = {{SocioLens project}},
  title = {On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks.},
  year = {2026},
  date = {2026-10-16},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld01-sl-c22/v1.0/}},
  note = {Claim LD01-SL-C22, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 652e6386fc3f14c05493e2524a1e1c8c78a0702a154fbde562794667e5616f53.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD01-SL-C22 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-16; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 16 October 2026 to build the LD-01 debate; a new Claim, first version, written from the project's LD-01 source notes, checked against full texts, official web pages or official abstracts. Block “ENTRY 22 — LD01-SL-C22” SHA-256 a1eda34206cb224f7fd5dd9ca26a6048633966ff7aadf98d7ffdfef47bfe3b2b; fixture fixtures/authored/claims/LD01-SL-C22--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD01-SL-C22 v1.0); the first version of a new Claim of the LD-01 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD01-SL-C22 v1.0" (Project Lead (Claude, AI), 2026-10-16); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 16 October 2026 to build the LD-01 debate; a new Claim, first version, written from the project's LD-01 source notes, checked against full texts, official web pages or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — basis: field 6 of ENTRY 22 — LD01-SL-C22 cites "Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024".