mlsanityData to 14 Oct 2026

Safety Assessment of Chinese Large Language Models

arXiv 2304.10436 ↗First posted 20 Apr 2023cs.CLcited by 85

Not on the map. Why?

Cited by 26 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the rapid popularity of large language models such as ChatGPT and GPT-4, a growing amount of attention is paid to their safety concerns. These models may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes such as fraud and dissemination of misleading information. Evaluating and enhancing their safety is particularly essential for the wide application of large language models (LLMs). To further promote the safe deployment of LLMs, we develop a Chinese LLM safety assessment benchmark. Our benchmark explores the comprehensive safety performance of LLMs from two perspectives: 8 kinds of typical safety scenarios and 6 types of more challenging instruction attacks. Our benchmark is based on a straightforward process in which it provides the test prompts and evaluates the safety of the generated responses from the evaluated model. In evaluation, we utilize the LLM's strong evaluation ability and develop it as a safety evaluator by prompting. On top of this benchmark, we conduct safety assessments and analyze 15 LLMs including the OpenAI GPT series and other well-known Chinese LLMs, where we observe some interesting findings. For example, we find that instruction attacks are more likely to expose safety issues of all LLMs. Moreover, to promote the development and deployment of safe, responsible, and ethical AI, we publicly release SafetyPrompts including 100k augmented prompts and responses by LLMs.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 13 of its 25 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

13 of 25 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  7. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  8. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  9. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  10. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  11. 2021EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Trainingcited by 34
  12. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
1 more
  1. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 33; 2025: 22; 2026: 8.

85 papers.

Sort:

By area:

  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  6. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  7. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  9. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  10. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  11. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  12. 2024Large Language Models Meet NLP: A Surveycited by 103
38 more
  1. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  2. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 88
  3. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  4. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  6. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  8. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 60
  9. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 53
  10. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  11. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  12. 2024Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attackscited by 38
  13. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 35
  14. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34
  15. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 34
  16. 2024On the Challenges and Opportunities in Generative AIcited by 31
  17. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 27
  18. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  19. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
  20. 2023Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Modelcited by 24
  21. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  22. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 21
  23. 2023Fairness of ChatGPTcited by 20
  24. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  25. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  26. 2024Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Modelscited by 19
  27. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  28. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  29. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 18
  30. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  31. 2023GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Modelscited by 16
  32. 2024CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Modelscited by 15
  33. 2023CLEVA: Chinese Language Models EVAluation Platformcited by 13
  34. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Modelscited by 13
  35. 2023A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Modelscited by 12
  36. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
  37. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  38. 2026VoxSafeBench: Not Just What Is Said, but Who, How, and Wherecited by 8

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinese
  2. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  3. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findings
  4. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMs
  5. 2025Safety and Security Analysis of Large Language Models: Benchmarking Risk Profile and Harm Potential
  6. 2026CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns
  7. 2024Large Language Model Safety: A Holistic Survey
  8. 2024OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety