Claim LD07-HQ-C02 (v1.1)
In a 2025 preprint that re-ran 37 annotation tasks from 21 published social-science studies with 18 language models (the Llama 3, Qwen 2.5 and 3, Gemma 3 and GPT-4o families), a plausible annotation configuration chosen at random led to a different conclusion from the one drawn from human ground-truth labels about 31% of the time for the lowest-risk models tested (GPT-4o, Qwen 2.5 32B and Llama 3.1 70B) and about half the time for the smallest, with higher risk near conventional significance thresholds.
Provenance and status
- Claim type
- Estimate
- Publication status
- Eligible for publication, with qualifiers: preprint, corpus
- Verification state
- Checked against the primary source; the source is a preprint; cleared with a source-specific qualifier (recorded as “CLEARED WITH QUALIFIER”). Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
- Does not license
- “31% of published findings are wrong”; future-model extrapolation
- Limitations
- Preprint; corpus/model-generation specific; configuration space defined by authors
- Source (as cited)
- Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2
- Source type
- Preprint (recorded as “Preprint”)
- Institutional position
- Independent academic
- Maturity of the evidence
- Rapidly maturing evidence
- Recorded date
- 22 October 2026
- Currentness / correction
- This is the current version. No correction or concern is recorded on its sources.
Sources this version cites
Version
This page shows version v1.1. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.
This pinned version is also the current version.
What changed in this version
This version supersedes version v1.0 (LD07-HQ-C02--v1.0), which stays available with its record unchanged.
- Changed fields
- 2 PUBLIC-FACING CLAIM; 12 MODEL / TOOL / VERSION; 14 MAIN RESULT; 25 RECORDED DATE
- What changed
- The claim no longer says “state-of-the-art models”, a phrase that ages into a claim about whatever models are current. It now names the four model families tested and the three lowest-risk models in Table 3, and says what the 31% measures: the chance that a plausible annotation configuration, chosen at random, leads to a different conclusion from the one drawn from human ground-truth labels. The model field lists the families and sizes, and the main result gives the per-model figures and names Table 3. The figures, the source and the other fields are unchanged.
- Why
- An external research-integrity review of the site on 22 October 2026 (an AI reviewer, not a person) found the figures correct but asked that the tested models be named and Table 3 cited, because “state-of-the-art” reads as a claim about current models. SocioLens re-read arXiv:2509.08825 v2 on 22 October 2026. Its §3 lists the 18 models as the “Llama 3 family (1B-70B)”, the “Qwen 2.5 and 3 families (1.5B-72B)”, the “Gemma family (1B-27B)” and “GPT-4o variants”; Table 3 gives an LLM hacking risk of 0.312 for GPT-4o, 0.315 for Qwen2.5-32B, 0.316 for Llama-3.1-70B and 0.318 for Qwen2.5-72B; the risk is defined as the probability of a wrong conclusion when a configuration is picked at random from the plausible ones, averaging Type I and Type II errors; and §5.2 says the risk “ranges from 31% for the best-performing 70B parameter models to 50% for 1B parameter models”, with conclusions compared against those from ground-truth annotations. The check and this revision were done by an AI agent reading the preprint, not by a person.
Raw record: every recorded field, with its coded values
All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.
- Claim type (coded)
- ESTIMATE (as printed: “ESTIMATE”)
- Publication status (coded)
- ELIGIBLE (PREPRINT, CORPUS) (as printed: “ELIGIBLE WITH PREPRINT/CORPUS QUALIFIER”)
- Verification state (coded)
- CONTENT_VERIFIED_PRIMARY (PREPRINT, SOURCE_SPECIFIC)
- Source type (coded)
- PREPRINT
- Maturity (coded)
- RAPIDLY_MATURING
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD07-HQ-C02 | — |
| 2 | PUBLIC-FACING CLAIM | In a 2025 preprint that re-ran 37 annotation tasks from 21 published social-science studies with 18 language models (the Llama 3, Qwen 2.5 and 3, Gemma 3 and GPT-4o families), a plausible annotation configuration chosen at random led to a different conclusion from the one drawn from human ground-truth labels about 31% of the time for the lowest-risk models tested (GPT-4o, Qwen 2.5 32B and Llama 3.1 70B) and about half the time for the smallest, with higher risk near conventional significance thresholds. | — |
| 3 | CLAIM TYPE | ESTIMATE | base_types: ESTIMATE |
| 4 | RESEARCH STRAND | A | — |
| 5 | MATURITY | RAPIDLY MATURING | states: RAPIDLY_MATURING |
| 6 | SOURCE | Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2 | source_ids: SRC-LD07-BAUMANN-LLM-HACKING-2025 |
| 7 | SOURCE TYPE | Preprint | classes: PREPRINT |
| 8 | INSTITUTIONAL POSITION | Independent academic | — |
| 9 | SCOPE | 37 CSS annotation tasks from 21 studies | — |
| 10 | DESIGN | Large multi-study re-annotation/configuration audit | — |
| 11 | SAMPLE / DATASET | ~13M labels; 2,361 realistic hypotheses | — |
| 12 | MODEL / TOOL / VERSION | 18 instruction-tuned models from four families: Llama 3 (1B–70B), Qwen 2.5 and Qwen 3 (1.5B–72B), Gemma 3 (1B–27B), GPT-4o and GPT-4o-mini; exact versions in the preprint’s Appendix C.1 | — |
| 13 | COMPARATOR | Study-specific ground-truth/reference annotation and hypothesis conclusions | — |
| 14 | MAIN RESULT | Probability that a randomly chosen plausible annotation configuration yields a conclusion different from the ground-truth conclusion (“LLM hacking risk”, Type I and Type II errors averaged): about 31% for the lowest-risk models (GPT-4o 31.2%, Qwen2.5-32B 31.5%, Llama-3.1-70B 31.6%; Qwen2.5-72B 31.8%; Table 3), about 50% for 1B-parameter models; risk higher near p=.05 | — |
| 15 | INFERENTIAL TARGET | Preservation of statistical conclusions under annotation configuration choice | — |
| 16 | DOES NOT LICENSE | “31% of published findings are wrong”; future-model extrapolation | — |
| 17 | VERIFICATION STATE | CLEARED WITH QUALIFIER | state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT, SOURCE_SPECIFIC · verified_by: UNKNOWN |
| 18 | LIMITATIONS | Preprint; corpus/model-generation specific; configuration space defined by authors | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | Egami; Yang; Li; Nakamura qualifies task-level agreement interpretation | — |
| 20 | GEOGRAPHY / LANGUAGE | Mostly English CSS | — |
| 21 | DISTRIBUTIONAL LEDGER | No broad subgroup guarantee | — |
| 22 | R01 / FIELD-MAP LINKS | T1, D9, R-1, R-2 | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Peer-reviewed replication with current models and substantially lower error | — |
| 24 | PUBLICATION STATUS | ELIGIBLE WITH PREPRINT/CORPUS QUALIFIER | state: ELIGIBLE · qualifiers: PREPRINT, CORPUS |
| 25 | RECORDED DATE | 2026-10-22 | — |
Field 22 references resolved against Field Map v0.2: T1 (TENSION); D9 (DOMAIN); R-1 (LENS); R-2 (LENS)
The sources as recorded, with their record IDs and coded source types:
Pinned Sources (field 6, version-specific)
- “Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2” → Baumann et al., “Large Language Model Hacking” (SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0) — Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2; class PREPRINT (explicit public provenance pin)
Cite this page
SocioLens project. (2026, October 22). In a 2025 preprint that re-ran 37 annotation tasks from 21 published social-science studies with 18 language models (the Llama 3, Qwen 2.5 and 3, Gemma 3 and GPT-4o families), a plausible annotation configuration chosen at random led to a different conclusion from the one drawn from human ground-truth labels about 31% of the time for the lowest-risk models tested (GPT-4o, Qwen 2.5 32B and Llama 3.1 70B) and about half the time for the smallest, with higher risk near conventional significance thresholds [Claim LD07-HQ-C02, version v1.1]. SocioLens. https://sociolens.eu/claims/ld07-hq-c02/v1.1/
@misc{sociolens_ld07_hq_c02_v1_1,
author = {{SocioLens project}},
title = {In a 2025 preprint that re-ran 37 annotation tasks from 21 published social-science studies with 18 language models (the Llama 3, Qwen 2.5 and 3, Gemma 3 and GPT-4o families), a plausible annotation configuration chosen at random led to a different conclusion from the one drawn from human ground-truth labels about 31\% of the time for the lowest-risk models tested (GPT-4o, Qwen 2.5 32B and Llama 3.1 70B) and about half the time for the smallest, with higher risk near conventional significance thresholds.},
year = {2026},
date = {2026-10-22},
howpublished = {\url{https://sociolens.eu/claims/ld07-hq-c02/v1.1/}},
note = {Claim LD07-HQ-C02, version v1.1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 9b4fb6d2e633ea89503bef785aa2bcdab93ef37baaf2f6acfe34c0051c452591.
Repository-authored inputs (not Drive exports)
- SocioLens claim revision — LD07-HQ-C02 v1.1 (v1.1) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-22; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and docs/PROJECT.md "Research revisions" (a published claim changes only through a new version that states what changed and why). Block “ENTRY 02 — LD07-HQ-C02” SHA-256
384d5f6ef2a0be7225536b0739876851ce580ed950bcee1c2a4ad7ff5906d056; fixturefixtures/authored/claim-revisions/LD07-HQ-C02--v1.1.
Builder decisions (not Drive data)
- VERSION = v1.1 — basis: Claim version taken from the repository-authored revision fixture version (SocioLens claim revision — LD07-HQ-C02 v1.1); a new version of LD07-HQ-C02 v1.0, which stays routable (AGENTS.md research integrity rule 3)
- CURRENT_POINTER = true — basis: repository-authored revision "SocioLens claim revision — LD07-HQ-C02 v1.1" (Project Lead (Claude, AI), 2026-10-22); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and docs/PROJECT.md "Research revisions" (a published claim changes only through a new version that states what changed and why)
- FIELD_17_CONTEXT_QUALIFIERS = PREPRINT, SOURCE_SPECIFIC — basis: R03 — HQ Cross-Batch Synthesis & Adjudication (v0.1, Drive 1V-HOVaPFwF8BTf60-LGPsJXf59V3kRlQ)
- FIELD_6_SOURCE_REFERENCE = SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — basis: field 6 of ENTRY 02 — LD07-HQ-C02 cites "Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2"; the field-6 text of this ClaimVersion's repository-authored ledger block; the same Source record LD07-HQ-C02 v1.0 cites.
- SOURCE_PROVENANCE_PIN = SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2".