Research opportunity · AI in the research apparatus
Does post-training change what an LLM measures when it codes text?
In short: has this been done?
Related work: Partly addressed. Related studies cover parts of this question; what they leave open is set out below. Only a first-pass search has been done, so it is not yet known whether a fuller search would find a study that does the whole design.
How far checked: First-pass screen only. A first-pass AI web search on 12 October 2026. No deeper check yet. Every search and check so far was done by AI; no person or domain expert has checked them yet.
The searches, the nearest work found and what the search could not reach
Earlier version (v1)
This is RO-LD07-05 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
Holding base weights fixed, how much do instruction tuning, preference tuning and region-specific fine-tunes shift coding validity and downstream estimates on stance and attitude annotation tasks, and do the shifts track the fine-tune's target culture?
Gap type: Causal mechanism not isolatedOne PhD chapter or paperLow costRelated work: Partly addressed
Why it matters
Observed differences between providers are routinely attributed to alignment or regulation, but existing comparisons cannot separate those from differences in base models and training data. Within-family checkpoint contrasts can isolate the effect of post-training on measurement itself.
The debate lists this among what we still do not know: How much do jurisdictional alignment regimes causally change social-science measurement, rather than merely refusal behavior?
Evidence that motivates it
- Provider and jurisdictional context can be associated with materially different refusal and accuracy patterns on politically sensitive prompts; existing evidence does not isolate regulation as the causal mechanism. (LD07-HQ-C11 v1.0)
How this study uses it: Provider and jurisdiction are associated with different response patterns, but the causal mechanism is not isolated. - Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents. (LD07-HQ-C08 v1.0)
How this study uses it: Chinese-language silicon-sample studies show cross-model differences whose source is unknown.
Each claim is shown as it read when this version was published.
A possible design
Use model families with public checkpoints at each post-training stage (base, supervised fine-tuning, preference tuning) and community or regional fine-tunes of the same base. Run established stance and attitude coding tasks with expert gold labels in two or more languages. Estimate the stage effect on accuracy, bias by target group and downstream coefficients, holding prompts and decoding fixed.
Data needed
Open checkpoints with documented training stages; annotated political-text corpora in at least two languages; GPU time.
Feasibility
One PhD chapter or paper. Low: open-weight checkpoints and existing labelled corpora.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
For survey simulation, Ziaei (22 September 2026) traces base-to-tuned checkpoints and finds supervised tuning roughly halves response dispersion; Santurkar et al. contrast base and RLHF opinions. No study found applies the same checkpoint design to annotation and coding validity or to regional fine-tunes of a shared base.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Ziaei, "The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance," arXiv:2609.25760 (2026)
Base-to-SFT-to-DPO checkpoint trace on World Values Survey simulation; survey simulation only, not coding tasks. - Santurkar et al., "Whose Opinions Do Language Models Reflect?" ICML 2023
Base versus RLHF opinion shift on OpinionsQA; opinions, not measurement validity. - Kotte, "Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science," arXiv:2606.12426 (2026)
Audits social-desirability bias in three 7B instruction-tuned annotators; no base-model contrast.
Search queries used
base model versus instruction-tuned public opinion survey responses alignment effect social science measurement same model familyeffect of RLHF instruction tuning on LLM text annotation stance detection coding accuracy base vs chat model comparisonpost-training stage checkpoints OLMo Tulu SFT DPO effect on annotation codingregion-specific fine-tune same base model political attitudes values survey
Cite this page
SocioLens project. (2026, October 12). Does post-training change what an LLM measures when it codes text? [Research opportunity RO-LD07-05, version v1]. SocioLens. https://sociolens-web.pages.dev/opportunities/ro-ld07-05/v1/
@misc{sociolens_ro_ld07_05_v1,
author = {{SocioLens project}},
title = {Does post-training change what an LLM measures when it codes text?},
year = {2026},
date = {2026-10-12},
howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-05/v1/}},
note = {Research opportunity RO-LD07-05, version v1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Provenance
RO-LD07-05 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.