Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7% were of closed commercial models, and 42% involved a model already retired when the paper was published or scheduled to retire within two years.

SocioLens project,

Claim LD06-SL-C29 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

How many published AI evaluations outside medicine can no longer be repeated?In biomedical publications, 42% of mentions of the 50 most used models involved a model retired by publication or due to retire within two years; median 538 days to retirement.

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: preprint
Verification state: Checked against the abstract of the primary source only. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A rate for fields other than biomedicine
Limitations: Read in abstract only; very recent preprint; an extraction agent read the abstracts
Source (as cited): Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026
Source type: Preprint (recorded as “Preprint”)
Institutional position: Independent academic
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026 (Preprint)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (PREPRINT) (as printed: “ELIGIBLE WITH PREPRINT QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (ABSTRACT_ONLY)
Source type (coded): PREPRINT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C29	—
2	PUBLIC-FACING CLAIM	Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7% were of closed commercial models, and 42% involved a model already retired when the paper was published or scheduled to retire within two years.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	C — Evaluation in governance, audits and red-teaming	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026	source_ids: SRC-SL-WOLFRATH-ET-AL-MODEL-RETIREMENT-ARXIV-2609-04699
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	Independent academic	—
9	SCOPE	PubMed original research applying a specific language model to a biomedical task	—
10	DESIGN	Agent-assisted extraction from abstracts, validated on a subset; lifecycle data for the top 50 models	—
11	SAMPLE / DATASET	61,077 abstracts screened; 5,242 publications	—
12	MODEL / TOOL / VERSION	The 50 most frequently used models	—
13	COMPARATOR	Providers’ retirement dates	—
14	MAIN RESULT	77.7% closed-weight commercial; 42% retired by publication or within two years; median 538 days from publication to retirement	—
15	INFERENTIAL TARGET	How far published evaluations of closed models can be reproduced later	—
16	DOES NOT LICENSE	A rate for fields other than biomedicine	—
17	VERIFICATION STATE	CONTENT VERIFIED ABSTRACT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: ABSTRACT_ONLY · verified_by: UNKNOWN
18	LIMITATIONS	Read in abstract only; very recent preprint; an extraction agent read the abstracts	—
19	CONTRADICTORY / SUPPORTING SOURCES	Bears on the Field Map unknown of independent evaluation under API retirement	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Replication in other fields; evidence on actual re-run failures	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026” → Wolfrath et al. (2026), “Model Retirement Creates…” (SRC-SL-WOLFRATH-ET-AL-MODEL-RETIREMENT-ARXIV-2609-04699--v1.0) — Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026; class PREPRINT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7% were of closed commercial models, and 42% involved a model already retired when the paper was published or scheduled to retire within two years [Claim LD06-SL-C29, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c29/v1.0/

@misc{sociolens_ld06_sl_c29_v1_0,
  author = {{SocioLens project}},
  title = {Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7\% were of closed commercial models, and 42\% involved a model already retired when the paper was published or scheduled to retire within two years.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c29/v1.0/}},
  note = {Claim LD06-SL-C29, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 f470a1500e21d14933473e9999c29504a9df120746f3016225efbb54c8aaec60.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C29 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 29 — LD06-SL-C29” SHA-256 e20cd94d4bbac28e9f949f729b460c39c746f8f41a6fe6759800826307136d98; fixture fixtures/authored/claims/LD06-SL-C29--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C29 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C29 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-WOLFRATH-ET-AL-MODEL-RETIREMENT-ARXIV-2609-04699--v1.0 — basis: field 6 of ENTRY 29 — LD06-SL-C29 cites "Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-WOLFRATH-ET-AL-MODEL-RETIREMENT-ARXIV-2609-04699--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026".