# COLD: A Benchmark for Chinese Offensive Language Detection

arXiv: https://arxiv.org/abs/2201.06025 · HTML: https://mlsanity.com/paper/2201.06025/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 70 references that are in the map.
- First posted on arXiv 16 Jan 2022.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Does Gender Matter? Towards Fairness in Dialogue Systems](https://mlsanity.com/paper/1910.10486/) (2019)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [A Large-Scale Chinese Short-Text Conversation Dataset](https://mlsanity.com/paper/2008.03946/) (2020)
- [On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark](https://mlsanity.com/paper/2110.08466/) (2021)
- [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://mlsanity.com/paper/2012.00413/) (2020)
- [Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts](https://mlsanity.com/paper/2108.11830/) (2021)
- [Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00699/) (2020)
- [SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection](https://mlsanity.com/paper/2108.03070/) (2021)
- [EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training](https://mlsanity.com/paper/2108.01547/) (2021)
- [A systematic review of Hate Speech automatic detection using Natural Language Processing](https://mlsanity.com/paper/2106.00742/) (2021)
- [Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases](https://mlsanity.com/paper/2112.07868/) (2021)
- [Assessing Political Prudence of Open-domain Chatbots](https://mlsanity.com/paper/2106.06157/) (2021)

## Built on by (most cited first, 30 of 76)

- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Self-Guard: Empower the LLM to Safeguard Itself](https://mlsanity.com/paper/2310.15851/) (2023)
- [ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors](https://mlsanity.com/paper/2402.16444/) (2024)
- [Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarks](https://mlsanity.com/paper/2305.04446/) (2023)
- [Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis](https://mlsanity.com/paper/2308.16705/) (2023)
- [Flames: Benchmarking Value Alignment of LLMs in Chinese](https://mlsanity.com/paper/2311.06899/) (2023)
- [Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks](https://mlsanity.com/paper/2202.08011/) (2022)
- [Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?](https://mlsanity.com/paper/2308.15399/) (2023)
- [EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training](https://mlsanity.com/paper/2203.09313/) (2022)
- [Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning](https://mlsanity.com/paper/2310.03400/) (2023)
- [MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili](https://mlsanity.com/paper/2408.03468/) (2024)
- [KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Application](https://mlsanity.com/paper/2305.17701/) (2023)
- [Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals](https://mlsanity.com/paper/2308.15192/) (2023)
- [CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models](https://mlsanity.com/paper/2306.16244/) (2023)
- [SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks](https://mlsanity.com/paper/2502.11090/) (2025)
- [Cross-Cultural Transfer Learning for Chinese Offensive Language Detection](https://mlsanity.com/paper/2303.17927/) (2023)
- [Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Features](https://mlsanity.com/paper/2310.06458/) (2023)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human](https://mlsanity.com/paper/2304.07849/) (2023)
- [CLEVA: Chinese Language Models EVAluation Platform](https://mlsanity.com/paper/2308.04813/) (2023)
- [Quite Good, but Not Enough: Nationality Bias in Large Language Models -- A Case Study of ChatGPT](https://mlsanity.com/paper/2405.06996/) (2024)
- [SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singapore](https://mlsanity.com/paper/2405.01842/) (2024)
- [Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings](https://mlsanity.com/paper/2505.24341/) (2025)
- [Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement](https://mlsanity.com/paper/2507.07640/) (2025)
- [PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model](https://mlsanity.com/paper/2203.17090/) (2022)
- [Collaborative decoding of critical tokens for boosting factuality of large language models](https://mlsanity.com/paper/2402.17982/) (2024)
- [OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety](https://mlsanity.com/paper/2403.12316/) (2024)
- [STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection](https://mlsanity.com/paper/2501.15451/) (2025)

## Abstract (arXiv)

Offensive language detection is increasingly crucial for maintaining a civilized social media platform and deploying pre-trained language models. However, this task in Chinese is still under exploration due to the scarcity of reliable datasets. To this end, we propose a benchmark --COLD for Chinese offensive language analysis, including a Chinese Offensive Language Dataset --COLDATASET and a baseline detector --COLDETECTOR which is trained on the dataset. We show that the COLD benchmark contributes to Chinese offensive language detection which is challenging for existing resources. We then deploy the COLDETECTOR and conduct detailed analyses on popular Chinese pre-trained language models. We first analyze the offensiveness of existing generative models and show that these models inevitably expose varying degrees of offensive issues. Furthermore, we investigate the factors that influence the offensive generations, and we find that anti-bias contents and keywords referring to certain groups or revealing negative attitudes trigger offensive outputs easier.