# SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinese

arXiv: https://arxiv.org/abs/2310.05818 · HTML: https://mlsanity.com/paper/2310.05818/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 7 of 7)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility](https://mlsanity.com/paper/2307.09705/) (2023)

## Built on by (most cited first, 17 of 17)

- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents](https://mlsanity.com/paper/2311.11797/) (2023)
- [ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation](https://mlsanity.com/paper/2405.14125/) (2024)
- [Fake Alignment: Are LLMs Really Aligned Well?](https://mlsanity.com/paper/2311.05915/) (2023)
- [SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks](https://mlsanity.com/paper/2502.11090/) (2025)
- [AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic](https://mlsanity.com/paper/2403.09017/) (2024)
- [CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2406.10311/) (2024)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values](https://mlsanity.com/paper/2501.07071/) (2025)
- [MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?](https://mlsanity.com/paper/2507.19598/) (2025)
- [AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field](https://mlsanity.com/paper/2509.18776/) (2025)
- [TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis](https://mlsanity.com/paper/2505.24672/) (2025)
- [FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios](https://mlsanity.com/paper/2605.00706/) (2026)

## Abstract (arXiv)

Large language models (LLMs), like ChatGPT and GPT-4, have demonstrated remarkable abilities in natural language understanding and generation. However, alongside their positive impact on our daily tasks, they can also produce harmful content that negatively affects societal perceptions. To systematically assess the safety of Chinese LLMs, we introduce SuperCLUE-Safety (SC-Safety) - a multi-round adversarial benchmark with 4912 open-ended questions covering more than 20 safety sub-dimensions. Adversarial human-model interactions and conversations significantly increase the challenges compared to existing methods. Experiments on 13 major LLMs supporting Chinese yield the following insights: 1) Closed-source models outperform open-sourced ones in terms of safety; 2) Models released from China demonstrate comparable safety levels to LLMs like GPT-3.5-turbo; 3) Some smaller models with 6B-13B parameters can compete effectively in terms of safety. By introducing SC-Safety, we aim to promote collaborative efforts to create safer and more trustworthy LLMs. The benchmark and findings provide guidance on model selection. Our benchmark can be found at https://www.CLUEbenchmarks.com