Claim LD07-HQ-C05 (v1.1)
LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.
Research opportunities this claim motivates
What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.
- Can synthetic respondents recover strong treatment-effect heterogeneity?Forecasting the pattern of average effects works in some US archives but overstates magnitudes.
- Do silicon samples work outside the North Atlantic, and in which language?Forecasting success comes from US archives; the claim would change with failure on non-US archives.
Provenance and status
- Claim type
- Fact
- Publication status
- Eligible for publication, with qualifiers: estimand
- Verification state
- Checked against the primary source; cleared with a source-specific qualifier (recorded as “CLEARED WITH QUALIFIER”). Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
- Does not license
- Synthetic population substitution; unbiased effect sizes; mechanism recovery; subgroup validity; non-US transport
- Limitations
- US survey-experiment archive; the correlations in the main result are taken from the authors’ accepted manuscript (posted on OSF), not checked against the Nature version of record
- Source (as cited)
- Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x
- Source type
- Peer-reviewed journal article (recorded as “Peer-reviewed journal article”)
- Institutional position
- Independent academic; one author also industry-affiliated
- Maturity of the evidence
- Contested evidence (positive bounded use)
- Recorded date
- 21 October 2026
- Currentness / correction
- This is the current version. No correction or concern is recorded on its sources.
Sources this version cites
Version
This is the current version, v1.1. Earlier versions stay available, and their records never change.
What changed in this version
This version supersedes version v1.0 (LD07-HQ-C05--v1.0), which stays available with its record unchanged.
- Changed fields
- 14 MAIN RESULT; 18 LIMITATIONS; 25 RECORDED DATE
- What changed
- The correlation is no longer held back. The main result now gives it: across 469 effects from 70 experiments, GPT-4-derived predictions correlated with the original effect sizes at r = 0.85 (95% CI 0.81–0.88), and at r = 0.92 (95% CI 0.89–0.94) after the authors’ adjustment for sampling error in the original estimates. The limitations now say that these figures come from the authors’ accepted manuscript, not from the Nature version of record. The public-facing claim, the source, the scope, the verification state and the publication status are unchanged; the verification state is the one recorded for version 1.0 and does not cover the two correlations, which were read in the manuscript only.
- Why
- Version 1.0 held back the exact r=.85 figure and its attribution to the Nature version of record (marked HOLD in the R03 LD-07 Canonical Claim Ledger v1.0), because the exact figure had not been extracted line by line from the Nature article. On 21 October 2026 the project owner supplied the authors’ accepted manuscript of the paper, as posted on OSF, which states that it has been accepted for publication in Nature. Its page 4, under "Primary test archive", reports: "In an analysis of 469 effects from 70 experiments, GPT-4-derived predictions were strongly correlated with original effect sizes (r = 0.85, 95% CI = [0.81, 0.88]; r_adj = 0.92, 95% CI = [0.89, 0.94])". The Nature version of record (656:115–122) was still not read, because the publisher blocks the project’s tools, so this version does not claim that the published article prints the same figures; the project owner chose to publish them with that attribution rather than wait for the version of record. The copy read was a print of the manuscript without a text layer, so the figures were read from the page image. The check and this revision were done by AI agents reading the manuscript, not by a person.
Raw record: every recorded field, with its coded values
All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.
- Claim type (coded)
- FACT (as printed: “FACT”)
- Publication status (coded)
- ELIGIBLE (ESTIMAND) (as printed: “ELIGIBLE WITH ESTIMAND QUALIFIER”)
- Verification state (coded)
- CONTENT_VERIFIED_PRIMARY (SOURCE_SPECIFIC)
- Source type (coded)
- PEER_REVIEWED_JOURNAL_ARTICLE
- Maturity (coded)
- CONTESTED (POSITIVE_BOUNDED_USE)
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD07-HQ-C05 | — |
| 2 | PUBLIC-FACING CLAIM | LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes. | — |
| 3 | CLAIM TYPE | FACT | base_types: FACT |
| 4 | RESEARCH STRAND | C — Synthetic respondents/agents | — |
| 5 | MATURITY | CONTESTED / POSITIVE BOUNDED USE | states: CONTESTED · qualifiers: POSITIVE_BOUNDED_USE |
| 6 | SOURCE | Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x | source_ids: SRC-LD07-ASHOKKUMAR-NATURE-2026 |
| 7 | SOURCE TYPE | Peer-reviewed journal article | classes: PEER_REVIEWED_JOURNAL_ARTICLE |
| 8 | INSTITUTIONAL POSITION | Independent academic; one author also industry-affiliated | — |
| 9 | SCOPE | US nationally representative text-based survey experiments | — |
| 10 | DESIGN | Forecast/model simulation benchmark against realized experimental effects and human forecasts | — |
| 11 | SAMPLE / DATASET | 70 experiments; 469 effects; 119,330 participants; 15 secondary megastudies/606 effects | — |
| 12 | MODEL / TOOL / VERSION | GPT-4 primary plus prominent open-weight models | — |
| 13 | COMPARATOR | Realized human treatment effects; pooled human/expert forecasts | — |
| 14 | MAIN RESULT | Strong effect-pattern correlation: across 469 effects from 70 experiments, GPT-4-derived predictions correlated with the original effect sizes at r = 0.85 (95% CI 0.81–0.88; adjusted r = 0.92, 95% CI 0.89–0.94, correcting for sampling error in the original estimates); human-forecast-comparable accuracy; systematic magnitude overestimation | — |
| 15 | INFERENTIAL TARGET | Aggregate effect forecasting / replication triage | — |
| 16 | DOES NOT LICENSE | Synthetic population substitution; unbiased effect sizes; mechanism recovery; subgroup validity; non-US transport | — |
| 17 | VERIFICATION STATE | CLEARED WITH QUALIFIER | state: CONTENT_VERIFIED_PRIMARY · qualifiers: SOURCE_SPECIFIC · verified_by: UNKNOWN |
| 18 | LIMITATIONS | US survey-experiment archive; the correlations in the main result are taken from the authors’ accepted manuscript (posted on OSF), not checked against the Nature version of record | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | Li & Ji shows realism ≠ ATE estimation; Cui/Li/Zhou shows null→significant inflation | — |
| 20 | GEOGRAPHY / LANGUAGE | US / English | — |
| 21 | DISTRIBUTIONAL LEDGER | Subgroup performance unresolved | — |
| 22 | R01 / FIELD-MAP LINKS | D9, R-1, T1 | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Failure on independent non-US experimental archives | — |
| 24 | PUBLICATION STATUS | ELIGIBLE WITH ESTIMAND QUALIFIER | state: ELIGIBLE · qualifiers: ESTIMAND |
| 25 | RECORDED DATE | 2026-10-21 | — |
Field 22 references resolved against Field Map v0.2: D9 (DOMAIN); R-1 (LENS); T1 (TENSION)
The sources as recorded, with their record IDs and coded source types:
Pinned Sources (field 6, version-specific)
- “Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x” → Ashokkumar et al. (2026), “Large language models can…” (SRC-LD07-ASHOKKUMAR-NATURE-2026--v1.0) — Ashokkumar, Hewitt, Ghezae & Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026); class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)
Cite this page
SocioLens project. (2026, October 21). LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes [Claim LD07-HQ-C05, version v1.1]. SocioLens. https://sociolens-web.pages.dev/claims/ld07-hq-c05/v1.1/
@misc{sociolens_ld07_hq_c05_v1_1,
author = {{SocioLens project}},
title = {LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.},
year = {2026},
date = {2026-10-21},
howpublished = {\url{https://sociolens-web.pages.dev/claims/ld07-hq-c05/v1.1/}},
note = {Claim LD07-HQ-C05, version v1.1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 c288810c1fbdf4d0f1675f5ed9becf269d6c3bb81d6cebf78d93f7c1a04f234e.
Repository-authored inputs (not Drive exports)
- SocioLens claim revision — LD07-HQ-C05 v1.1 (v1.1) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-21; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and docs/PROJECT.md "Research revisions" (a published claim changes only through a new version that states what changed and why). Block “ENTRY 05 — LD07-HQ-C05” SHA-256
2238d5e25e9b47450ea1e5b56dd054a5cd5b4b6596760b75f24f0617d1120026; fixturefixtures/authored/claim-revisions/LD07-HQ-C05--v1.1.
Builder decisions (not Drive data)
- VERSION = v1.1 — basis: Claim version taken from the repository-authored revision fixture version (SocioLens claim revision — LD07-HQ-C05 v1.1); a new version of LD07-HQ-C05 v1.0, which stays routable (AGENTS.md research integrity rule 3)
- CURRENT_POINTER = true — basis: repository-authored revision "SocioLens claim revision — LD07-HQ-C05 v1.1" (Project Lead (Claude, AI), 2026-10-21); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and docs/PROJECT.md "Research revisions" (a published claim changes only through a new version that states what changed and why)
- FIELD_17_CONTEXT_QUALIFIERS = SOURCE_SPECIFIC — basis: R03 — HQ Cross-Batch Synthesis & Adjudication (v0.1, Drive 1V-HOVaPFwF8BTf60-LGPsJXf59V3kRlQ)
- FIELD_6_SOURCE_REFERENCE = SRC-LD07-ASHOKKUMAR-NATURE-2026--v1.0 — basis: field 6 of ENTRY 05 — LD07-HQ-C05 cites "Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x"; the field-6 text of this ClaimVersion's repository-authored ledger block; the same Source record LD07-HQ-C05 v1.0 cites.
- SOURCE_PROVENANCE_PIN = SRC-LD07-ASHOKKUMAR-NATURE-2026--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x".