Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks
Not on the map. Why?
Abstract (arXiv)
In Chinese content governance, large language models are increasingly used to identify toxic and discriminatory expressions, yet both task operationalization and ironic expression can alter bias measurement readings. We conducted a preregistered multi-task behavioral audit of multiple Chinese large language models. In the content classification experiment, the core panel's classification error rate rose from 10.9% in the formal register to 23.5% in the yinyang (an echoic-irony register) version, a risk difference of +12.7 percentage points. (95% CI [+9.7, +15.7]). Inter-rater agreement was lower in the yinyang register (majority-vote kappa = 0.299), so the increase contains a component of label uncertainty propagation, but the direction of increase was consistent under both adjudicated and majority-vote reference standards. Explicit irony-aware prompts reduced the intervention panel's error rate by 9.6 percentage points. We operationalized the 29 shared proposition anchors into three task forms: social consensus estimation (Form A), high-stakes decision (Form B), and controlled association rating (Form C); independent manipulation checks found that all 58 A-C comparison units received unanimous agreement, whereas the low A-B convergence was confirmed by calibrated majority adjudication (87.8% supporting the minority judgment); on the irony-probe materials, the interactions of B and C relative to A were +0.577 and +1.121, respectively. Thus, when evaluating large language models for Chinese content governance, the effect of task operationalization on bias readings is an empirical question that requires data to answer, and safety evaluation should incorporate the ironic register as a test condition.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 10 Oct 2026.
Builds on
21 of 38 references.
References · 21
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023LIMA: Less Is More for Alignmentcited by 1,100
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
- 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
- 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
9 more
- 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
- 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
- 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
- 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 76
- 2023Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarkscited by 46
- 2024ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbationscited by 31
- 2023CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Modelscited by 28
- 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflectioncited by 27
- 2025Irony in Emojis: A Comparative Study of Human and LLM Interpretationcited by 6
Built on by
Not cited yet.