mlsanityData to 7 Oct 2026

SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks

arXiv 2502.11090 ↗First posted 16 Feb 2025cs.CL cs.AIcited by 23

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

With the rapid advancement of Large Language Models (LLMs), the safety of LLMs has been a critical concern requiring precise assessment. Current benchmarks primarily concentrate on single-turn dialogues or a single jailbreak attack method to assess the safety. Additionally, these benchmarks have not taken into account the LLM's capability of identifying and handling unsafe information in detail. To address these issues, we propose a fine-grained benchmark SafeDialBench for evaluating the safety of LLMs across various jailbreak attacks in multi-turn dialogues. Specifically, we design a two-tier hierarchical safety taxonomy that considers 6 safety dimensions and generates more than 4000 multi-turn dialogues in both Chinese and English under 22 dialogue scenarios. We employ 7 jailbreak attack strategies, such as reference attack and purpose reverse, to enhance the dataset quality for dialogue generation. Notably, we construct an innovative assessment framework of LLMs, measuring capabilities in detecting, and handling unsafe information and maintaining consistency when facing jailbreak attacks. Experimental results across 17 LLMs reveal that Yi-34B-Chat and GLM4-9B-Chat demonstrate superior safety performance, while Llama3.1-8B-Instruct and o3-mini exhibit safety vulnerabilities.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 44 of its 66 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

44 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usability14 Feb
  2. 2024Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks1 Jul
  3. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  4. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue27 Feb
  5. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models21 Feb
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023Mistral 7Bcited by 3,065
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
25 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  5. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  6. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  7. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  8. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  9. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  10. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  11. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  12. 2023A Survey on Fairness in Large Language Modelscited by 91
  13. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  14. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  15. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  16. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
  17. 2023Privacy Issues in Large Language Models: A Surveycited by 57
  18. 2024Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perceptioncited by 55
  19. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Modelscited by 55
  20. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  21. 2024MoralBench: Moral Evaluation of LLMscited by 50
  22. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  23. 2023SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinesecited by 17
  24. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  25. 2025Causal Information Prioritization for Efficient Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Taskscited by 45
  2. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  3. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  4. 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Studycited by 12
  5. 2025Automating Deception: Scalable Multi-Turn LLM Jailbreakscited by 7
  6. 2026JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attackscited by 6
  7. 2025Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energycited by 4
  8. 2025SafeMT: Multi-turn Safety for Multimodal Language Modelscited by 4
  9. 2025Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detectioncited by 4
  10. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  11. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  12. 2026VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Modelscited by 3
11 more
  1. 2026SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Cachingcited by 3
  2. 2026SafeDream: Safety World Model for Proactive Early Jailbreak Detectioncited by 2
  3. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  4. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1
  5. 2026JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Modelscited by 1
  6. 2025PsyLite Technical Reportcited by 0
  7. 2026Defensive M2S: Training Guardrail Models on Compressed Multi-turn Conversationscited by 0
  8. 2026SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamicscited by 0
  9. 2026Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detectioncited by 0
  10. 2026Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCHcited by 0
  11. 2026EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agentscited by 0