Does post-training change what an LLM measures when it codes text?

Research question

Holding base weights fixed, how much do instruction tuning, preference tuning and region-specific fine-tunes shift coding validity and downstream estimates on stance and attitude annotation tasks, and do the shifts track the fine-tune's target culture?

Why it matters

Observed differences between providers are routinely attributed to alignment or regulation, but existing comparisons cannot separate those from differences in base models and training data. Within-family checkpoint contrasts can isolate the effect of post-training on measurement itself.

The debate lists this among what we still do not know: How much do jurisdictional alignment regimes causally change social-science measurement, rather than merely refusal behavior?

Evidence that motivates it

1. Provider and jurisdictional context can be associated with materially different refusal and accuracy patterns on politically sensitive prompts; existing evidence does not isolate regulation as the causal mechanism. (LD07-HQ-C11 v1.0)
   Provider and jurisdiction are associated with different response patterns, but the causal mechanism is not isolated.
   Cites: Pan & Xu (2026), “Political censorship in large language…”
2. Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents. (LD07-HQ-C08 v1.0)
   Chinese-language silicon-sample studies show cross-model differences whose source is unknown.
   Cites: Gong Weigang & Huang Siyuan (2025), “整体事实还是有偏样本——基于大语言模型生成…”; Ling Wanying & Cui Sizhan (2025), “复现的限度——‘硅基样本’的可能与可为”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Use model families with public checkpoints at each post-training stage (base, supervised fine-tuning, preference tuning) and community or regional fine-tunes of the same base. Run established stance and attitude coding tasks with expert gold labels in two or more languages. Estimate the stage effect on accuracy, bias by target group and downstream coefficients, holding prompts and decoding fixed.

Data needed

Open checkpoints with documented training stages; annotated political-text corpora in at least two languages; GPU time.

Feasibility

One PhD chapter or paper. Low: open-weight checkpoints and existing labelled corpora.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

For survey simulation, Ziaei (22 September 2026) traces base-to-tuned checkpoints and finds supervised tuning roughly halves response dispersion; Santurkar et al. contrast base and RLHF opinions; a MENA study compares alignment stages for cultural-value survey responses, and a September 2026 preprint finds task faithfulness eroding most at the DPO stage. No study found applies the checkpoint design to annotation and coding validity against expert gold labels, or to regional fine-tunes of a shared base.

Literature check. Six new queries found no study that holds base weights fixed and measures how SFT, preference tuning and regional fine-tunes shift coding validity and downstream estimates on stance or attitude annotation. Checkpoint traces now exist for survey simulation (Ziaei 2026), for cultural-value survey responses across alignment stages in MENA countries (Zahraei et al., arXiv:2510.13154) and for task faithfulness, which they find is amplified during DPO (Zahraei et al., arXiv:2610.00568). None of these tests annotation validity against expert gold labels or regional fine-tunes of a shared base. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Abstracts only for the new preprints; full texts may contain annotation-style tasks not visible in the abstracts. The Chinese-language query returned general explainers and one alphaXiv page on factual sycophancy, not annotation studies. Santurkar et al. was not re-checked. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Ziaei, "The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance," arXiv:2609.25760 (2026)
  Checkpoint trace from base model through SFT, DPO and GRPO stages on World Values Survey simulation; survey simulation only, not coding tasks.
• Santurkar et al., "Whose Opinions Do Language Models Reflect?" ICML 2023
  Base versus RLHF opinion shift on OpinionsQA; opinions, not measurement validity.
• Kotte, "Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science," arXiv:2606.12426 (2026)
  Audits social-desirability bias in three 7B instruction-tuned annotators; no base-model contrast.
• Zahraei, Tur, Hakkani-Tür & Asgari, "The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs," arXiv:2510.13154 (v1 15 October 2025; v2 18 June 2026) Judged from an abstract, summary or search listing only
  Abstract read: 26 models compared with 1.53M survey responses in 16 MENA countries; internal distributions match human data better than free-text answers ('suppression'), with comparisons across alignment stages and a DPO-stage feature in Tulu-3-8B. Survey-response alignment, not annotation validity against gold labels; does not test regional fine-tunes of a shared base, per the abstract.
• Zahraei, Singh, Tur & Hakkani-Tür, "Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness," arXiv:2610.00568 (30 September 2026) Judged from an abstract, summary or search listing only
  Abstract read: intermediate post-training checkpoints show models increasingly override task instructions, with the gap growing most and becoming least visible at the DPO stage, on a FaithConflict dataset. Shows a stage effect on task faithfulness that a coding-validity study would expect; the abstract does not mention annotation, classification or stance coding.
• "Evaluating Decision Models for Text Annotation in Computational Social Science," arXiv:2609.24574 (2026; authors not confirmed) Judged from an abstract, summary or search listing only Some details not confirmed
  Abstract read: compares a commercial decision model and open-weight counterparts against 19 LLMs on zero-shot annotation; no base versus tuned checkpoints of the same model are mentioned.

References

Sources behind the motivating claims:

• Pan & Xu, “Political censorship in large language models originating from China,” PNAS Nexus 5(2):pgag013 (2026) (SRC-LD07-PAN-XU-CENSORSHIP-2026, original)
• Gong Weigang & Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025) (SRC-LD07-GONG-HUANG-CGSS-2025, original)
• Ling Wanying & Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025) (SRC-LD07-LING-CUI-SILICON-SAMPLES-2025, no checked link to the original yet)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_05_v2,
  author = {{SocioLens project}},
  title = {Does post-training change what an LLM measures when it codes text?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-05/v2/}},
  note = {Research opportunity RO-LD07-05, version v2}
}

@misc{ro_ld07_05_nearest_1,
  title = {Ziaei, "The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance," arXiv:2609.25760 (2026)},
  url = {https://arxiv.org/abs/2609.25760},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_05_nearest_2,
  title = {Santurkar et al., "Whose Opinions Do Language Models Reflect?" ICML 2023},
  url = {https://arxiv.org/abs/2303.17548},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_05_nearest_3,
  title = {Kotte, "Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science," arXiv:2606.12426 (2026)},
  url = {https://arxiv.org/abs/2606.12426},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_05_nearest_4,
  title = {Zahraei, Tur, Hakkani-Tür \& Asgari, "The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs," arXiv:2510.13154 (v1 15 October 2025; v2 18 June 2026)},
  url = {https://arxiv.org/abs/2510.13154},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld07_05_nearest_5,
  title = {Zahraei, Singh, Tur \& Hakkani-Tür, "Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness," arXiv:2610.00568 (30 September 2026)},
  url = {https://arxiv.org/abs/2610.00568},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld07_05_nearest_6,
  title = {"Evaluating Decision Models for Text Annotation in Computational Social Science," arXiv:2609.24574 (2026; authors not confirmed)},
  url = {https://arxiv.org/abs/2609.24574},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only; Some details not confirmed}
}

@misc{src_ld07_pan_xu_censorship_2026,
  title = {Pan \& Xu, “Political censorship in large language models originating from China,” PNAS Nexus 5(2):pgag013 (2026)},
  url = {https://doi.org/10.1093/pnasnexus/pgag013},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_gong_huang_cgss_2025,
  title = {Gong Weigang \& Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025)},
  url = {https://www.xsyk021.com/article/id/4c1886eb-c385-4019-92ff-d1cd75547fa5},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_ling_cui_silicon_samples_2025,
  title = {Ling Wanying \& Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025)},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}