In a randomized trial run from February to June 2025 with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19% (confidence interval 2% to 39%), although the developers estimated it had cut their time by 20%; METR later called the data from a follow-up begun in August 2025, and reported in February 2026, an unreliable signal because of selection.

SocioLens project,

Claim LD04-SL-C07 (v1.1)

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: preprint
Verification state: Checked against the primary source; the source is a preprint. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A general claim that AI slows experts; any estimate for current tools
Limitations: Very small sample; one kind of setting; tools have changed; METR believes developers are now more sped up, and its own 2026 redesign post treats later estimates as unreliable
Source (as cited): Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025; METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026
Source type: Preprint, Independent research report (recorded as “Preprint + research nonprofit post”)
Institutional position: Independent research nonprofit
Maturity of the evidence: Contested evidence
Recorded date: 22 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Version

This page shows version v1.1. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.

This pinned version is also the current version.

Versions: v1.0, v1.1 · current

What changed in this version

This version supersedes version v1.0 (LD04-SL-C07--v1.0), which stays available with its record unchanged.

Changed fields: 2 PUBLIC-FACING CLAIM; 14 MAIN RESULT; 25 RECORDED DATE
What changed: The public-facing claim now gives the confidence interval of the 19% slowdown (2% to 39% longer), the dates of the trial (February to June 2025), and says the follow-up began in August 2025 and was reported in February 2026, instead of “its 2026 follow-up data”. The main result dates METR’s assessment to February 2026 and the follow-up to August 2025. The figures, the sources and the other fields are unchanged.
Why: An external research-integrity review of the site on 22 October 2026 (an AI reviewer, not a person) found the figures correct but asked for the uncertainty interval beside the 19% and for the follow-up to be dated by when it began. SocioLens re-read both sources on 22 October 2026. The preprint (arXiv:2507.09089) tests “AI tools at the February–June 2025 frontier”. METR’s post of 24 February 2026 says the early-2025 study “found the use of AI causes tasks to take 19% longer, with a confidence interval between +2% and +39%”, that “we started a new experiment in August 2025”, and that its data give “an unreliable signal” because of selection. The check and this revision were done by an AI agent reading the sources, not by a person.

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (PREPRINT) (as printed: “ELIGIBLE WITH PREPRINT QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT)
Source type (coded): PREPRINT, INDEPENDENT_RESEARCH_REPORT
Maturity (coded): CONTESTED

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD04-SL-C07	—
2	PUBLIC-FACING CLAIM	In a randomized trial run from February to June 2025 with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19% (confidence interval 2% to 39%), although the developers estimated it had cut their time by 20%; METR later called the data from a follow-up begun in August 2025, and reported in February 2026, an unreliable signal because of selection.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Performance while the tool is in use	—
5	MATURITY	CONTESTED	states: CONTESTED
6	SOURCE	Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025; METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026	source_ids: SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089, SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24
7	SOURCE TYPE	Preprint + research nonprofit post	classes: PREPRINT, INDEPENDENT_RESEARCH_REPORT
8	INSTITUTIONAL POSITION	Independent research nonprofit	—
9	SCOPE	Mature open-source repositories; developers with about five years on their projects; early-2025 tools	—
10	DESIGN	Randomized controlled trial at task level	—
11	SAMPLE / DATASET	16 developers, 246 tasks	—
12	MODEL / TOOL / VERSION	Early-2025 AI coding tools	—
13	COMPARATOR	Tasks on which AI use was not allowed	—
14	MAIN RESULT	Completion time +19% (CI +2% to +39%) with AI allowed; developers estimated a 20% reduction; in February 2026 METR described the data from its follow-up, begun in August 2025, as only very weak evidence because developers unwilling to work without AI opted out and others withheld tasks	—
15	INFERENTIAL TARGET	Assisted speed of highly experienced developers on familiar codebases	—
16	DOES NOT LICENSE	A general claim that AI slows experts; any estimate for current tools	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Very small sample; one kind of setting; tools have changed; METR believes developers are now more sped up, and its own 2026 redesign post treats later estimates as unreliable	—
19	CONTRADICTORY / SUPPORTING SOURCES	Contrasts with large speed gains in lab coding tasks; consistent with meta-analytic evidence that controlled-experiment gains shrink in real settings	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	The most experienced may gain least, or lose, on their own familiar work	—
22	R01 / FIELD-MAP LINKS	LD-04, D2	—
23	WHAT WOULD CHANGE THIS CLAIM?	A larger, well-selected trial with current tools	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT
25	RECORDED DATE	2026-10-22	—

Field 22 references resolved against Field Map v0.2: LD-04 (LIVING_DEBATE); D2 (DOMAIN)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025” → Becker et al. (2025), “Measuring the Impact of Early-2025…” (SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089--v1.0) — Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025; class PREPRINT (explicit public provenance pin)
“METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026” → METR (2026), “We are Changing our Developer Productivity…” (SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24--v1.0) — METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026; class INDEPENDENT_RESEARCH_REPORT (explicit public provenance pin)

Cite this page

SocioLens project. (2026, October 22). In a randomized trial run from February to June 2025 with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19% (confidence interval 2% to 39%), although the developers estimated it had cut their time by 20%; METR later called the data from a follow-up begun in August 2025, and reported in February 2026, an unreliable signal because of selection [Claim LD04-SL-C07, version v1.1]. SocioLens. https://sociolens.eu/claims/ld04-sl-c07/v1.1/

@misc{sociolens_ld04_sl_c07_v1_1,
  author = {{SocioLens project}},
  title = {In a randomized trial run from February to June 2025 with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19\% (confidence interval 2\% to 39\%), although the developers estimated it had cut their time by 20\%; METR later called the data from a follow-up begun in August 2025, and reported in February 2026, an unreliable signal because of selection.},
  year = {2026},
  date = {2026-10-22},
  howpublished = {\url{https://sociolens.eu/claims/ld04-sl-c07/v1.1/}},
  note = {Claim LD04-SL-C07, version v1.1}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 1a1bacd89dd7cd5e0a07432090575181c0cb6f66ba6a1007705d7e0322637300.

Repository-authored inputs (not Drive exports)

SocioLens claim revision — LD04-SL-C07 v1.1 (v1.1) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-22; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and docs/PROJECT.md "Research revisions" (a published claim changes only through a new version that states what changed and why). Block “ENTRY 07 — LD04-SL-C07” SHA-256 1903c38281604566272369bfd31373e0903589b91c502ab1f2ae96d67461e4dd; fixture fixtures/authored/claim-revisions/LD04-SL-C07--v1.1.

Builder decisions (not Drive data)

VERSION = v1.1 — basis: Claim version taken from the repository-authored revision fixture version (SocioLens claim revision — LD04-SL-C07 v1.1); a new version of LD04-SL-C07 v1.0, which stays routable (AGENTS.md research integrity rule 3)
CURRENT_POINTER = true — basis: repository-authored revision "SocioLens claim revision — LD04-SL-C07 v1.1" (Project Lead (Claude, AI), 2026-10-22); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and docs/PROJECT.md "Research revisions" (a published claim changes only through a new version that states what changed and why)
FIELD_6_SOURCE_REFERENCE = SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089--v1.0 — basis: field 6 of ENTRY 07 — LD04-SL-C07 cites "Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025"; the first field-6 segment of this ClaimVersion's repository-authored ledger block; the same Source record LD04-SL-C07 v1.0 cites.
FIELD_6_SOURCE_REFERENCE = SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24--v1.0 — basis: field 6 of ENTRY 07 — LD04-SL-C07 cites "METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026"; the second field-6 segment of this ClaimVersion's repository-authored ledger block; the same Source record LD04-SL-C07 v1.0 cites.
SOURCE_PROVENANCE_PIN = SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025".
SOURCE_PROVENANCE_PIN = SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026".