Research opportunity · The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them?
How many published AI evaluations outside medicine can no longer be repeated?
In short: has this been done?
Related work: Partly addressed. Related studies cover parts of this question; what they leave open is set out below. Only a first-pass search has been done, so it is not yet known whether a fuller search would find a study that does the whole design.
How far checked: First-pass screen only. A first-pass AI web search on 15 October 2026. No deeper check yet. Every search and check so far was done by AI; no person or domain expert has checked them yet.
The searches, the nearest work found and what the search could not reach
Follow this opportunity: Atom feed of RO-LD06-06 updates (new versions, and revisions of the claims it rests on).
Question
Among published studies in NLP, the social sciences and education research that evaluate closed language models through an API, what share used a model version that has since been retired or is due to retire, and how many of a sample can still be re-run to reproduce their main result?
Gap type: Untested beyond the original settingMaster's thesisLow costRelated work: Partly addressed
Why it matters
Independent evaluation depends on others being able to repeat a result. In biomedicine, 42% of mentions of the most used models involved one retired by publication or due within two years. If the same holds in the social sciences, a large part of the published evidence on what language models do, including much of this platform's evidence base, cannot be checked once the model is gone.
The debate lists this among what we still do not know: How far can published evaluations of closed models be repeated after the model is retired? (Field Map: independent evaluation under API retirement.)
Evidence that motivates it
- Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7% were of closed commercial models, and 42% involved a model already retired when the paper was published or scheduled to retire within two years.
How this study uses it: In biomedical publications, 42% of mentions of the 50 most used models involved a model retired by publication or due to retire within two years; median 538 days to retirement. - The 2025 Foundation Model Transparency Index found that the average transparency score of major AI developers fell from 58 in 2024 to 40 out of 100, and that no company adequately disclosed how far its training data overlap with the evaluations it is tested on; the authors note that changed indicators and companies may contribute to the decline.
How this study uses it: Developers give little detail for external reproduction of their evaluations.
A possible design
Sample papers from the ACL Anthology and from social-science and education journals (2022-2026) that evaluate API models; extract the model and version reported; match each to providers' deprecation records; estimate the share retired by publication or within two years, by field. For a random subsample with released code and data, attempt re-runs with the original model where still available and with its designated successor otherwise, and report how often the main result holds.
Data needed
Paper metadata and full texts; providers' public deprecation pages and archived copies; API budget for re-runs.
Feasibility
Master's thesis. Low; public papers, archived deprecation pages and a small API budget. Many papers do not report the exact version, which is itself a finding; extraction by a model must be validated by people, as in the biomedical study.
Proposal pack: this page and its references on one printable page, with BibTeX.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).
The biomedical version of the count exists, and software engineering has a re-run study of conference papers that used OpenAI models. No study was found that counts retirement exposure across NLP, social-science and education papers and re-runs a subsample.
How strong this evidence is
- Who searched and checked: no checker is recorded beyond the first-pass screen, which was part of the AI drafting (Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate). Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 4 of the 4 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (3 queries) and one arXiv query on the date shown; the ACL Anthology, Scopus, SocArXiv and education venues were not searched directly, and two 2026 workshop papers seen only in listings were not read. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- N. Wolfrath et al., "Model Retirement Creates Reproducibility Risk in Biomedical AI Publications," arXiv:2609.04699 (2026) Judged from an abstract, summary or search listing only
Abstract read for the LD-06 debate (LD06-SL-C29). The template design, limited to biomedicine and with no re-runs. - Angermeir, Amougou, Kreitz, Bauer et al., "Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies," arXiv:2510.25506 (2025) Judged from an abstract, summary or search listing only
Abstract read. Of 18 ICSE and ASE 2024 studies with artefacts using OpenAI models, 5 could be executed and none fully reproduced; the role of deprecated models was seen in search results only. - Balloccu, Schmidtová, Lango & Dušek, "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs," arXiv:2402.03927 (2024) Judged from an abstract, summary or search listing only
Abstract read. Review of NLP papers using closed models; its figures on how often versions are reported were seen in search results only. - Cheng, Mayya & Sedoc, "To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation," arXiv:2412.14461 (2024) Judged from an abstract, summary or search listing only
Abstract read. Names model retirement as a core threat to annotation reproducibility; does not count affected papers.
Search queries used
deprecated OpenAI models reproducibility published NLP papers model versions no longer available studyACL papers closed LLM versions reported snapshot reproducibility audit "model version" GPT-3.5 deprecated survey of paperssocial science research using ChatGPT API reproducibility model deprecation share of studies cannot be replicated retired model versionsarXiv: abs:deprecation AND abs:reproducibility
Cite this page
SocioLens project. (2026, October 15). How many published AI evaluations outside medicine can no longer be repeated? [Research opportunity RO-LD06-06, version v1]. SocioLens. https://sociolens-web.pages.dev/opportunities/ro-ld06-06/v1/
@misc{sociolens_ro_ld06_06_v1,
author = {{SocioLens project}},
title = {How many published AI evaluations outside medicine can no longer be repeated?},
year = {2026},
date = {2026-10-15},
howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-06/v1/}},
note = {Research opportunity RO-LD06-06, version v1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Provenance
RO-LD06-06 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 2026-10-15. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.