On 18 real tasks from two open-source repositories, an AI agent's code passed the maintainers' tests 38% of the time, but none of the 15 pull requests METR reviewed by hand could be merged as is; METR roughly estimated 26 minutes of human repair even for those that passed, about a third of the original human time.

SocioLens project,

Claim LD03-SL-C12 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

How much of the time AI saves its users do colleagues spend checking and fixing?None of the 15 agent pull requests METR reviewed by hand was mergeable as is, with repair estimated at about a third of the original human time even for those that passed the tests.
Is the time AI saves paid for with weaker checking?Even agent-written code that passed tests needed about 26 minutes of human repair, a third of the original human time, on METR's rough estimate.

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with a qualifier
Verification state: Checked against the primary source. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A general rate for current agents; an estimate of developer productivity
Limitations: Very small sample; repair times are rough subjective estimates by METR, not by the maintainers; one older model
Source (as cited): Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025
Source type: Independent research report (recorded as “Research nonprofit post”)
Institutional position: Independent research nonprofit
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025 (Independent research report)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY
Source type (coded): INDEPENDENT_RESEARCH_REPORT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD03-SL-C12	—
2	PUBLIC-FACING CLAIM	On 18 real tasks from two open-source repositories, an AI agent's code passed the maintainers' tests 38% of the time, but none of the 15 pull requests METR reviewed by hand could be merged as is; METR roughly estimated 26 minutes of human repair even for those that passed, about a third of the original human time.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	C — Work reorganization and verification	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025	source_ids: SRC-SL-REIN-METR-ALGORITHMIC-HOLISTIC-2025-08-13
7	SOURCE TYPE	Research nonprofit post	classes: INDEPENDENT_RESEARCH_REPORT
8	INSTITUTIONAL POSITION	Independent research nonprofit	—
9	SCOPE	Two open-source repositories; tasks taking humans 20 minutes to four hours	—
10	DESIGN	Small evaluation of agent output against tests and manual review	—
11	SAMPLE / DATASET	18 tasks; 15 pull requests reviewed manually	—
12	MODEL / TOOL / VERSION	An agent using Claude 3.7 Sonnet	—
13	COMPARATOR	Maintainer-written tests versus holistic review	—
14	MAIN RESULT	38% (±19%) pass maintainer tests; none of the 15 reviewed pull requests mergeable as is; estimated repair time 42 minutes on average and 26 minutes for test-passing pull requests	—
15	INFERENTIAL TARGET	The gap between passing automated checks and producing work an organization can use	—
16	DOES NOT LICENSE	A general rate for current agents; an estimate of developer productivity	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	Very small sample; repair times are rough subjective estimates by METR, not by the maintainers; one older model	—
19	CONTRADICTORY / SUPPORTING SOURCES	Consistent with persistent quality costs after AI adoption in open-source projects	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-03, D2	—
23	WHAT WOULD CHANGE THIS CLAIM?	Larger evaluations with current agents and maintainer-made repair estimates	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-03 (LIVING_DEBATE); D2 (DOMAIN)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025” → Rein (2025), “Research Update: Algorithmic vs. Holistic…” (SRC-SL-REIN-METR-ALGORITHMIC-HOLISTIC-2025-08-13--v1.0) — Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025; class INDEPENDENT_RESEARCH_REPORT (explicit public provenance pin)

Cited by Living Debate versions

Organizational absorption: Why micro gains do not become macro outcomes (v0.1) LD-03

Cite this page

SocioLens project. (2026, October 15). On 18 real tasks from two open-source repositories, an AI agent's code passed the maintainers' tests 38% of the time, but none of the 15 pull requests METR reviewed by hand could be merged as is; METR roughly estimated 26 minutes of human repair even for those that passed, about a third of the original human time [Claim LD03-SL-C12, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld03-sl-c12/v1.0/

@misc{sociolens_ld03_sl_c12_v1_0,
  author = {{SocioLens project}},
  title = {On 18 real tasks from two open-source repositories, an AI agent's code passed the maintainers' tests 38\% of the time, but none of the 15 pull requests METR reviewed by hand could be merged as is; METR roughly estimated 26 minutes of human repair even for those that passed, about a third of the original human time.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld03-sl-c12/v1.0/}},
  note = {Claim LD03-SL-C12, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 5e3fa5d5b2ce328f76c7b2b4ab914767dcd320a271ab506b81c78cbbdfc32883.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD03-SL-C12 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-03 debate; a new Claim, first version, written from the project's LD-03 source notes, checked against full texts or official abstracts. Block “ENTRY 12 — LD03-SL-C12” SHA-256 495c637b9c6c1f8216df23fb98de0a3f8576ca7414de54d8959f01b7ee5055ad; fixture fixtures/authored/claims/LD03-SL-C12--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD03-SL-C12 v1.0); the first version of a new Claim of the LD-03 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD03-SL-C12 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-03 debate; a new Claim, first version, written from the project's LD-03 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-REIN-METR-ALGORITHMIC-HOLISTIC-2025-08-13--v1.0 — basis: field 6 of ENTRY 12 — LD03-SL-C12 cites "Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-REIN-METR-ALGORITHMIC-HOLISTIC-2025-08-13--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Rein, “Research Update: Algorithmic vs. Holistic Evaluation,” METR, 13 August 2025".