# CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns

arXiv: https://arxiv.org/abs/2601.00588 · HTML: https://mlsanity.com/paper/2601.00588/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 2 Jan 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness](https://mlsanity.com/paper/2411.03350/) (2024)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [MiniCPM4: Ultra-Efficient LLMs on End Devices](https://mlsanity.com/paper/2506.07900/) (2025)
- [Flames: Benchmarking Value Alignment of LLMs in Chinese](https://mlsanity.com/paper/2311.06899/) (2023)
- [Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle](https://mlsanity.com/paper/2407.13833/) (2024)
- [ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations](https://mlsanity.com/paper/2406.12223/) (2024)
- [From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets](https://mlsanity.com/paper/2404.17874/) (2024)
- [FORTRESS: Frontier Risk Evaluation for National Security and Public Safety](https://mlsanity.com/paper/2506.14922/) (2025)
- [Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings](https://mlsanity.com/paper/2505.24341/) (2025)
- [Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement](https://mlsanity.com/paper/2507.07640/) (2025)
- [Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models](https://mlsanity.com/paper/2412.15265/) (2024)
- [Adversarial Text Normalization](https://mlsanity.com/paper/2206.04137/) (2022)
- [Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluation](https://mlsanity.com/paper/2503.06519/) (2025)
- [Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Models](https://mlsanity.com/paper/2502.19883/) (2025)
- [Fine-Grained Chinese Hate Speech Understanding: Span-Level Resources, Coded Term Lexicon, and Enhanced Detection Frameworks](https://mlsanity.com/paper/2507.11292/) (2025)
- [Libra: Large Chinese-based Safeguard for AI Content](https://mlsanity.com/paper/2507.21929/) (2025)

## Built on by (most cited first, 5 of 5)

- [A Comparative Evaluation of AI Agent Security Guardrails](https://mlsanity.com/paper/2604.24826/) (2026)
- [Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs](https://mlsanity.com/paper/2606.10487/) (2026)
- [JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data](https://mlsanity.com/paper/2605.24414/) (2026)
- [Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese](https://mlsanity.com/paper/2605.29667/) (2026)
- [Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges](https://mlsanity.com/paper/2609.01210/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in cost-sensitive and on-device scenarios, and safety guardrails have advanced mainly in English. However, real-world Chinese malicious queries typically conceal intent via homophones, pinyin, symbol-based splitting, and other Chinese-specific patterns. These Chinese-specific adversarial patterns create the safety evaluation gap that is not well captured by existing benchmarks focused on English. This gap is particularly concerning for lightweight models, which may be more vulnerable to such specific adversarial perturbations. To bridge this gap, we introduce the Chinese-Specific Safety Benchmark (CSSBench) that emphasizes these adversarial patterns and evaluates the safety of lightweight LLMs in Chinese. Our benchmark covers six domains that are common in real Chinese scenarios, including illegal activities and compliance, privacy leakage, health and medical misinformation, fraud and hate, adult content, and public and political safety, and organizes queries into multiple task types. We evaluate a set of popular lightweight LLMs and measure over-refusal behavior to assess safety-induced performance degradation. Our results show that the Chinese-specific adversarial pattern is a critical challenge for lightweight LLMs. This benchmark offers a comprehensive evaluation of LLM safety in Chinese, assisting robust deployments in practice.