Research opportunity · The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them?
Who builds and checks China's compliance test banks for generative AI?
In short: has this been done?
Related work: No study found. A first-pass search found no study that does the core design. It has not been checked in depth, so a fuller search could still find one.
How far checked: First-pass screen only. A first-pass AI web search on 15 October 2026. No deeper check yet. Every search and check so far was done by AI; no person or domain expert has checked them yet.
The searches, the nearest work found and what the search could not reach
This page shows RO-LD06-05 version v1, whose record never changes. It is the current version; if the opportunity is revised, the newer version appears at its main page.
Question
Under China's national standard GB/T 45654-2025, who builds the generated-content and refusal test banks that decide whether a generative AI service passes, how are the banks validated, and do providers' banks and third-party evaluators' banks give the same verdicts on the same models?
Gap type: Geographic or language coverageA PhD projectLow to medium costRelated work: No study found
Why it matters
China has written evaluation into compliance as fixed pass rates on test banks, a model unlike the EU's. The validity critiques of benchmarks (errors, contamination, unclear constructs) apply with force if providers build the banks they are scored on. Almost all evaluation research concerns English-language benchmarks and US, UK and EU institutions, so a study of the Chinese regime would also correct the field's geography.
The debate lists this among what we still do not know: Who builds the test banks under China's standard, and how are they checked?
Evidence that motivates it
- China’s national standard GB/T 45654-2025 for generative AI services sets a pass rate: at least 90% of sampled output must be free of 31 risk types; its reference appendix specifies a test bank of at least 2,000 questions, refusal of at least 95% of should-refuse questions, and refusal of no more than 5% of should-not-refuse questions. (LD06-SL-C25 v1.0)
How this study uses it: The standard requires a 90% qualified rate on sampled output and, in its reference appendix, test banks of at least 2,000 questions with refusal thresholds of 95% and 5%. - Re-annotating 5,700 questions across all 57 subjects of MMLU, a widely used knowledge benchmark, researchers estimated that 6.49% of its questions contain errors, 57% of those analysed in the Virology subset, and that the errors change how models rank. (LD06-SL-C04 v1.0)
How this study uses it: Benchmark items contain errors that change model rankings, a risk for any fixed bank. - An analysis of MMLU found that 28% of its questions require culturally sensitive knowledge and that 84.9% of its geography questions concern North America or Europe; model rankings change depending on whether the culturally sensitive questions are included, so machine-translated versions distort multilingual comparisons. (LD06-SL-C11 v1.0)
How this study uses it: Benchmark content reflects where it was made, which bears on comparing Chinese and Western safety testing.
Each claim is shown as it read when this version was published.
A possible design
A comparative study combining document analysis of published evaluation datasets and reports from Chinese third-party evaluation institutions, standard-setting bodies and providers; interviews with staff of evaluation institutions and providers; and, where banks or samples are public, an item audit for errors and overlap and a comparison of verdicts on the same open-weight models.
Data needed
Chinese-language documents and datasets (including any state-built benchmark datasets); interviewees in evaluation institutions; open-weight Chinese models for verdict comparisons.
Feasibility
A PhD project. Low to medium; researcher time, translation and some compute. Needs Chinese-language skills and a Chinese-literature search first; access to providers' banks is unlikely, so the design should rest on public datasets and interviews. Political sensitivity of the risk categories may limit what interviewees say.
Has this been done?
The search found no study that does the core design (first-pass screen of 15 October 2026; not yet checked in depth).
No study was found, in English or in the few Chinese results that surfaced, of who builds the test banks the standard requires, how they are validated, or whether providers' and third-party evaluators' banks agree. Existing work is the standard and its translation, academic safety benchmarks aligned with Chinese regulation, and one state-built benchmark dataset seen only in search listings.
How strong this evidence is
- Who searched and checked: no checker is recorded beyond the first-pass screen, which was part of the AI drafting (Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate). Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 3 of the 4 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (4 queries, one in Chinese) and one arXiv query on the date shown; CNKI, Wanfang and other Chinese databases could not be searched, so the absence of hits is especially weak evidence here. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- State Administration for Market Regulation & Standardization Administration of China, GB/T 45654-2025 (CSET English translation, 28 May 2026)
Full text read for the LD-06 debate (LD06-SL-C25). The primary text; says nothing about how banks are built or checked in practice. - CNCERT, "安全测评基准数据集1.0" (Safety evaluation benchmark dataset 1.0); publisher, date and URL not confirmed Judged from an abstract, summary or search listing onlySome details not confirmed
Judged from search listings; not read directly. Reportedly 3,000 questions covering the standard's 31 risk subcategories; a state-built bank and a key comparison object. - Zhang, Gao, Hu et al., "ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models," arXiv:2410.18491 (2024) Judged from an abstract, summary or search listing only
Abstract read. An academic benchmark aligned with Chinese content-moderation rules; does not study compliance banks. - Zhang, Kodama, Wu, Chen, Zhu & Hong, "Frontier Safety Policies for AI Emergency Preparedness in China," arXiv:2511.05526 (2025) Judged from an abstract, summary or search listing only
Abstract read. Policy analysis noting required pre-deployment assessments; does not study test banks.
Search queries used
GB/T 45654-2025 generative AI security test question bank evaluation studyChina generative AI filing CAC security assessment test question bank 2000 questions third-party evaluation study生成式人工智能服务安全基本要求 测试题库 第三方 评估 拒答率 研究study of Chinese AI safety evaluation institutions TC260-003 compliance testing providers third-party assessors keyword bank refusal rate analysisarXiv: abs:China AND abs:generative AND (filing OR standard OR TC260 OR regulation) AND abs:safety
Cite this page
SocioLens project. (2026, October 15). Who builds and checks China's compliance test banks for generative AI? [Research opportunity RO-LD06-05, version v1]. SocioLens. https://sociolens-web.pages.dev/opportunities/ro-ld06-05/v1/
@misc{sociolens_ro_ld06_05_v1,
author = {{SocioLens project}},
title = {Who builds and checks China's compliance test banks for generative AI?},
year = {2026},
date = {2026-10-15},
howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-05/v1/}},
note = {Research opportunity RO-LD06-05, version v1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Provenance
RO-LD06-05 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 2026-10-15. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.