Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis.

SocioLens project,

Claim LD07-HQ-C15 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

How do research ethics boards judge synthetic participants and AI-interpreted data?Agreement metrics do not establish that interpretation can be delegated, which is exactly what boards must weigh.
Does "LLMs can replace coders" depend on which humans you compare with?Human agreement is not a sufficient quality rule, so the choice of human reference matters.

Provenance and status

Claim type: Fact and Interpretation
Publication status: Eligible for publication, with a qualifier
Verification state: Checked against the primary source; the source is a preprint; cleared with a source-specific qualifier (recorded as “CLEARED WITH PREPRINT/DOMAIN QUALIFIER”). Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: Claim LLMs are better interpreters; one expert as universal ground truth
Limitations: Specific domains; Liu preprint; limited independent verifiers
Source (as cited): Liu et al., “Agreement Is Not Quality,” arXiv:2607.28890; PLOS Digital Health 2026
Source type: Preprint, Peer-reviewed journal article (recorded as “Preprint + peer-reviewed bounded case”)
Institutional position: Independent academic
Maturity of the evidence: Emerging evidence
Recorded date: 25 September 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): FACT + INTERPRETATION (as printed: “FACT + INTERPRETATION”)
Publication status (coded): ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT, SOURCE_SPECIFIC)
Source type (coded): PREPRINT, PEER_REVIEWED_JOURNAL_ARTICLE
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD07-HQ-C15	—
2	PUBLIC-FACING CLAIM	Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis.	—
3	CLAIM TYPE	FACT + INTERPRETATION	base_types: FACT, INTERPRETATION
4	RESEARCH STRAND	E	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Liu et al., “Agreement Is Not Quality,” arXiv:2607.28890; PLOS Digital Health 2026	source_ids: SRC-LD07-LIU-AGREEMENT-IS-NOT-QUALITY-2026, SRC-LD07-PLOS-THEMATIC-ANALYSIS-2026
7	SOURCE TYPE	Preprint + peer-reviewed bounded case	classes: PREPRINT, PEER_REVIEWED_JOURNAL_ARTICLE
8	INSTITUTIONAL POSITION	Independent academic	—
9	SCOPE	K-12 educator messages; healthcare thematic analysis	—
10	DESIGN	Blind independent expert verification + inductive comparison	—
11	SAMPLE / DATASET	Liu: 2,560 messages; 5 LLM systems; 3 humans; 855 blind pair comparisons	—
12	MODEL / TOOL / VERSION	Multiple LLM systems; PLOS contemporary frontier systems	—
13	COMPARATOR	Human coders and independent domain expert	—
14	MAIN RESULT	Human–LLM Jaccard .30 vs human–human .52, yet blind quality preference 51.5% human vs 48.5% LLM (p=.537); inductive coding in PLOS more variable than deductive	—
15	INFERENTIAL TARGET	Interpretive quality rather than mere agreement	—
16	DOES NOT LICENSE	Claim LLMs are better interpreters; one expert as universal ground truth	—
17	VERIFICATION STATE	CLEARED WITH PREPRINT/DOMAIN QUALIFIER	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT, SOURCE_SPECIFIC · verified_by: UNKNOWN
18	LIMITATIONS	Specific domains; Liu preprint; limited independent verifiers	—
19	CONTRADICTORY / SUPPORTING SOURCES	Than supports supervised augmentation	—
20	GEOGRAPHY / LANGUAGE	Primarily US/English	—
21	DISTRIBUTIONAL LEDGER	Shared human or model priors may both encode group-specific blind spots	—
22	R01 / FIELD-MAP LINKS	D3, D9, R-2	—
23	WHAT WOULD CHANGE THIS CLAIM?	Broad blinded expert replication of autonomous inductive analysis	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-09-25	—

Field 22 references resolved against Field Map v0.2: D3 (DOMAIN); D9 (DOMAIN); R-2 (LENS)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Liu et al., “Agreement Is Not Quality,” arXiv:2607.28890” → Liu et al., “Agreement Is Not Quality: Blind Expert…” (SRC-LD07-LIU-AGREEMENT-IS-NOT-QUALITY-2026--v1.0) — Liu et al., “Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth,” arXiv:2607.28890; class PREPRINT (explicit public provenance pin)
“PLOS Digital Health 2026” → “Large language models for thematic analysis in healthcare…” (SRC-LD07-PLOS-THEMATIC-ANALYSIS-2026--v1.0) — “Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts,” PLOS Digital Health 2026; class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)

Cited by Living Debate versions

AI in the research apparatus (v0.1) LD-07
AI in the research apparatus (v0.2) LD-07

Cite this page

SocioLens project. (2026, September 25). Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis [Claim LD07-HQ-C15, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld07-hq-c15/v1.0/

@misc{sociolens_ld07_hq_c15_v1_0,
  author = {{SocioLens project}},
  title = {Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis.},
  year = {2026},
  date = {2026-09-25},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld07-hq-c15/v1.0/}},
  note = {Claim LD07-HQ-C15, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 86724b97891c856c9fdbc18a994d499abc8675306704dfc44e86b1e018d7e58a.

Frozen Drive-derived inputs

R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED (v1.0) — Drive ID 1xVz_LrQAA8EA1tPhVWIrDgOUU3Zidlg9; revision: NOT_EXPOSED_BY_DRIVE_TEXT_EXPORT; Drive modified 2026-09-29T04:31:14.120Z; block “ENTRY 15 — LD07-HQ-C15” SHA-256 f4c50936c08e550987f95bd60f2fb38437ae11b9d6bbcddc6a706a2e7060e831; SocioLens — Programme Status & Authority Register v0.6 status “CURRENT.”; frozen fixture fixtures/frozen/r03-ld07-claim-ledger-v1.0-canonical.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the canonical ledger artifact version (R03 — LD-07 Canonical Claim Ledger v1.0); Implementation Architecture Selection §5
CURRENT_POINTER = true — basis: SocioLens — Programme Status & Authority Register v0.6 status for "R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED": "CURRENT."
FIELD_6_SOURCE_REFERENCE = SRC-LD07-LIU-AGREEMENT-IS-NOT-QUALITY-2026--v1.0 — basis: field 6 of ENTRY 15 — LD07-HQ-C15 cites "Liu et al., “Agreement Is Not Quality,” arXiv:2607.28890"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §9; ledger C15; KEY EVIDENCE.
FIELD_6_SOURCE_REFERENCE = SRC-LD07-PLOS-THEMATIC-ANALYSIS-2026--v1.0 — basis: field 6 of ENTRY 15 — LD07-HQ-C15 cites "PLOS Digital Health 2026"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §8; ledger C14/C15; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-LIU-AGREEMENT-IS-NOT-QUALITY-2026--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Liu et al., “Agreement Is Not Quality,” arXiv:2607.28890"; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §9; ledger C15; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-PLOS-THEMATIC-ANALYSIS-2026--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "PLOS Digital Health 2026"; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §8; ledger C14/C15; KEY EVIDENCE.