# A Chinese Dataset for Evaluating the Safeguards in Large Language Models

arXiv: https://arxiv.org/abs/2402.12193 · HTML: https://mlsanity.com/paper/2402.12193/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs](https://mlsanity.com/paper/2308.13387/) (2023)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [Diverse Adversaries for Mitigating Bias in Training](https://mlsanity.com/paper/2101.10001/) (2021)
- [BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B](https://mlsanity.com/paper/2311.00117/) (2023)
- [Goal-Oriented Prompt Attack and Safety Evaluation for LLMs](https://mlsanity.com/paper/2309.11830/) (2023)
- [Comprehensive Evaluation of ChatGPT Reliability Through Multilingual Inquiries](https://mlsanity.com/paper/2312.10524/) (2023)

## Built on by (most cited first, 12 of 12)

- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing](https://mlsanity.com/paper/2406.14230/) (2024)
- [ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models](https://mlsanity.com/paper/2410.18491/) (2024)
- [JailBench: A Comprehensive Chinese Security Assessment Benchmark for Large Language Models](https://mlsanity.com/paper/2502.18935/) (2025)
- [Auto-Prompt Ensemble for LLM Judge](https://mlsanity.com/paper/2510.06538/) (2025)
- [LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds](https://mlsanity.com/paper/2412.05232/) (2024)
- [Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting](https://mlsanity.com/paper/2503.01493/) (2025)
- [AnswerCarefully: A Dataset for Improving the Safety of Japanese LLM Output](https://mlsanity.com/paper/2506.02372/) (2025)
- [TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts](https://mlsanity.com/paper/2604.16542/) (2026)
- [Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction](https://mlsanity.com/paper/2609.25176/) (2026)
- [NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context](https://mlsanity.com/paper/2505.08734/) (2025)
- [Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats](https://mlsanity.com/paper/2606.26377/) (2026)

## Abstract (arXiv)

Many studies have demonstrated that large language models (LLMs) can produce harmful responses, exposing users to unexpected risks when LLMs are deployed. Previous studies have proposed comprehensive taxonomies of the risks posed by LLMs, as well as corresponding prompts that can be used to examine the safety mechanisms of LLMs. However, the focus has been almost exclusively on English, and little has been explored for other languages. Here we aim to bridge this gap. We first introduce a dataset for the safety evaluation of Chinese LLMs, and then extend it to two other scenarios that can be used to better identify false negative and false positive examples in terms of risky prompt rejections. We further present a set of fine-grained safety assessment criteria for each risk type, facilitating both manual annotation and automatic evaluation in terms of LLM response harmfulness. Our experiments on five LLMs show that region-specific risks are the prevalent type of risk, presenting the major issue with all Chinese LLMs we experimented with. Our data is available at https://github.com/Libr-AI/do-not-answer. Warning: this paper contains example data that may be offensive, harmful, or biased.