Do silicon samples work outside the North Atlantic, and in which language?

Research question

Using Afrobarometer, Arab Barometer and Latinobarómetro items fielded after model training cutoffs, does synthetic-sample accuracy at country, subgroup and individual level depend on interview language (for example Swahili or Hausa versus English, Arabic versus English), and do LLM codes of political constructs in low-resource languages pass a replacement test against trained in-country coders?

Why it matters

The validation studies in this debate come mainly from the US, Europe and China, and this screen found few from Africa, the Arab region or Latin America. Researchers working in and on those regions, where survey budgets are often tight and cheap substitutes most tempting, need evidence about their own contexts.

Evidence that motivates it

1. LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes. (LD07-HQ-C05 v1.1)
   Forecasting success comes from US archives; the claim would change with failure on non-US archives.
   Cites: Ashokkumar et al. (2026), “Large language models can…”
2. Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents. (LD07-HQ-C08 v1.0)
   Locally grounded validation (here, Chinese CGSS) finds representational bias and instability.
   Cites: Gong Weigang & Huang Siyuan (2025), “整体事实还是有偏样本——基于大语言模型生成…”; Ling Wanying & Cui Sizhan (2025), “复现的限度——‘硅基样本’的可能与可为”
3. Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity. (LD07-HQ-C06 v1.0)
   Subgroup and variance failures are documented in the contexts tested so far.
   Cites: Bisbee et al. (2024), “Synthetic Replacements for Human…”; Xie et al. (2026), “Evaluating the statistical realism of…”; Wang et al. NMI 2025; Gong Weigang & Huang Siyuan (2025), “整体事实还是有偏样本——基于大语言模型生成…”; Ling Wanying & Cui Sizhan (2025), “复现的限度——‘硅基样本’的可能与可为”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Select recent waves of two or three regional barometers. Generate synthetic samples conditioned on respondent profiles, prompting in English and in the interview language. Compare marginal distributions, subgroup estimates and key associations with the human data. Separately, have trained in-country coders and LLMs code a political-text task in one or two low-resource languages and apply a formal replacement test.

Data needed

Public barometer microdata; API and open-weight models with multilingual support; a small budget for in-country coders.

Feasibility

A PhD project. Low to medium: survey data are free; coder time is the main cost. Strongest with a co-author or partner institution in the region.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Studies use the World Values Survey and other global cultural surveys for a range of countries, including one comparing English with native-language prompting across ten countries (Liu et al., AAAI 2026) and fine-tuned models tested on unseen countries (Cao et al., NAACL 2025), plus Chilean and Egypt/US benchmarks, and NLP benchmarks such as AfroBench cover African languages. No study found benchmarks silicon samples against Afrobarometer, Arab Barometer or Latinobarometro by interview language, or validates low-resource-language coding of social-science constructs against trained in-country coders.

Literature check. Seven new queries (English, Spanish and Arabic; preprint- and journal-restricted; forward search on Liu, Kaneko & Chu) found no study that benchmarks silicon samples against Afrobarometer, Arab Barometer or Latinobarometro, or that compares interview-language with English prompting on those data. Cross-national simulation work stays on global cultural surveys (Cao et al., NAACL 2025; Liu et al., AAAI 2026) or general benchmarks (SimBench), and African-language LLM work is NLP benchmarking (AfroBench) rather than validation against trained in-country coders of social-science constructs. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Full texts of Cao et al. and SimBench were not read, so their dataset lists (and whether a regional barometer appears) are unconfirmed. Google Scholar, SSRN search and regional journals (e.g. African or Latin American political science journals) were not searched directly; the Spanish and Arabic queries returned only news and survey-organisation pages. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Qu & Wang, "Performance and biases of Large Language Models in public opinion simulation," Humanities & Social Sciences Communications 11:1095 (2024)
  World Values Survey in six countries including South Africa and Brazil, with native-language prompts for some countries; accuracy worse outside Western, English-speaking settings; regional barometers not used.
• González-Bustamante, Verelst & Cisternas, "Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case," arXiv:2509.09871 (2025)
  Chilean probability survey benchmark; item-level accuracy varies widely.
• AlKhamissi et al., "Investigating Cultural Alignment of Large Language Models," arXiv:2402.13231 (2024)
  Egypt/US survey simulation; alignment better when prompting in Arabic; one country pair.
• Kastrati et al., "Do language families matter?" Frontiers in Artificial Intelligence 9 (2026)
  LLM sentiment classification in 36 low-resource languages against prior models, not human coders or social-science constructs.
• Liu, Kaneko & Chu, "On the Alignment of Large Language Models with Global Human Opinion," arXiv:2509.01418 (2025); published in Proceedings of AAAI 2026
  World Values Survey framework; compares English with native-language prompting for 8 languages and 10 countries and finds language steering effective; no regional barometers, no validation against local coders.
• Cao, Liu, Arora, Augenstein, Röttger & Hershcovich, "Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations," NAACL 2025 (long), pp. 3141-3154 Some details not confirmed
  Fine-tunes LLMs on country-level response distributions from two global cultural surveys and tests on unseen questions, countries and one unseen survey; models still struggle on unseen questions. The abstract page does not name the surveys, the countries or the prompt language; it does not mention regional barometers, interview-language comparisons or coder validation (not confirmed from full text).
• Hu, Baumann, Lupo, Collier, Hovy & Röttger, "SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors," arXiv:2510.17516 (v4, Apr 2026)
  Benchmark of 20 datasets of human behaviour and opinion across a global participant pool; models struggle most with specific demographic groups. The abstract does not list the datasets, so whether any regional barometer is included was not confirmed; it is not a language-of-prompt or in-country-coder study.
• AfroBench, "AfroBench: How Good are Large Language Models on African Languages?" Findings of ACL 2025; arXiv:2311.07978 Judged from an abstract, summary or search listing only
  Seen in search results only (title, venue, identifier). An NLP benchmark of LLM performance on African-language tasks; not survey simulation and not validation of political-construct coding against trained in-country coders. Author list not checked.

References

Sources behind the motivating claims:

• Ashokkumar, Hewitt, Ghezae & Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026) (SRC-LD07-ASHOKKUMAR-NATURE-2026, original)
• Gong Weigang & Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025) (SRC-LD07-GONG-HUANG-CGSS-2025, original)
• Ling Wanying & Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025) (SRC-LD07-LING-CUI-SILICON-SAMPLES-2025, no checked link to the original yet)
• Bisbee, Clinton, Dorff, Kenkel & Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis 32(4):401–416 (2024) (SRC-LD07-BISBEE-SYNTHETIC-REPLACEMENTS-2024, original)
• Xie et al., “Evaluating the statistical realism of LLM-generated social science data,” PNAS 123(19):e2538145123 (2026) (SRC-LD07-XIE-SSDATABENCH-2026, original)
• Wang et al. NMI 2025 (SRC-R03-WANG-MORGENSTERN-DICKERSON-NMI-2025, no checked link to the original yet)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_08_v2,
  author = {{SocioLens project}},
  title = {Do silicon samples work outside the North Atlantic, and in which language?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-08/v2/}},
  note = {Research opportunity RO-LD07-08, version v2}
}

@misc{ro_ld07_08_nearest_1,
  title = {Qu \& Wang, "Performance and biases of Large Language Models in public opinion simulation," Humanities \& Social Sciences Communications 11:1095 (2024)},
  url = {https://www.nature.com/articles/s41599-024-03609-x},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_08_nearest_2,
  title = {González-Bustamante, Verelst \& Cisternas, "Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case," arXiv:2509.09871 (2025)},
  url = {https://arxiv.org/abs/2509.09871},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_08_nearest_3,
  title = {AlKhamissi et al., "Investigating Cultural Alignment of Large Language Models," arXiv:2402.13231 (2024)},
  url = {https://arxiv.org/abs/2402.13231},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_08_nearest_4,
  title = {Kastrati et al., "Do language families matter?" Frontiers in Artificial Intelligence 9 (2026)},
  url = {https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2026.1854873/full},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_08_nearest_5,
  title = {Liu, Kaneko \& Chu, "On the Alignment of Large Language Models with Global Human Opinion," arXiv:2509.01418 (2025); published in Proceedings of AAAI 2026},
  url = {https://arxiv.org/abs/2509.01418},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_08_nearest_6,
  title = {Cao, Liu, Arora, Augenstein, Röttger \& Hershcovich, "Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations," NAACL 2025 (long), pp. 3141-3154},
  url = {https://aclanthology.org/2025.naacl-long.162/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Some details not confirmed}
}

@misc{ro_ld07_08_nearest_7,
  title = {Hu, Baumann, Lupo, Collier, Hovy \& Röttger, "SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors," arXiv:2510.17516 (v4, Apr 2026)},
  url = {https://arxiv.org/abs/2510.17516},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_08_nearest_8,
  title = {AfroBench, "AfroBench: How Good are Large Language Models on African Languages?" Findings of ACL 2025; arXiv:2311.07978},
  url = {https://aclanthology.org/2025.findings-acl.976.pdf},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_ld07_ashokkumar_nature_2026,
  title = {Ashokkumar, Hewitt, Ghezae \& Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026)},
  url = {https://doi.org/10.1038/s41586-026-10742-x},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_gong_huang_cgss_2025,
  title = {Gong Weigang \& Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025)},
  url = {https://www.xsyk021.com/article/id/4c1886eb-c385-4019-92ff-d1cd75547fa5},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_ling_cui_silicon_samples_2025,
  title = {Ling Wanying \& Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025)},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_bisbee_synthetic_replacements_2024,
  title = {Bisbee, Clinton, Dorff, Kenkel \& Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis 32(4):401–416 (2024)},
  url = {https://doi.org/10.1017/pan.2024.5},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_xie_ssdatabench_2026,
  title = {Xie et al., “Evaluating the statistical realism of LLM-generated social science data,” PNAS 123(19):e2538145123 (2026)},
  url = {https://doi.org/10.1073/pnas.2538145123},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_r03_wang_morgenstern_dickerson_nmi_2025,
  title = {Wang et al. NMI 2025},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}