mlsanityData to 7 Oct 2026

SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal

arXiv 2406.14598 ↗First posted 20 Jun 2024cs.AIcited by 224

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Evaluating aligned large language models' (LLMs) ability to recognize and reject unsafe user requests is crucial for safe, policy-compliant deployments. Existing evaluation efforts, however, face three limitations that we address with SORRY-Bench, our proposed benchmark. First, existing methods often use coarse-grained taxonomies of unsafe topics, and are over-representing some fine-grained topics. For example, among the ten existing datasets that we evaluated, tests for refusals of self-harm instructions are over 3x less represented than tests for fraudulent activities. SORRY-Bench improves on this by using a fine-grained taxonomy of 44 potentially unsafe topics, and 440 class-balanced unsafe instructions, compiled through human-in-the-loop methods. Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations. We supplement SORRY-Bench with 20 diverse linguistic augmentations to systematically examine these effects. Third, existing evaluations rely on large LLMs (e.g., GPT-4) for evaluation, which can be computationally expensive. We investigate design choices for creating a fast, accurate automated safety evaluator. By collecting 7K+ human annotations and conducting a meta-evaluation of diverse LLM-as-a-judge designs, we show that fine-tuned 7B LLMs can achieve accuracy comparable to GPT-4 scale LLMs, with lower computational cost. Putting these together, we evaluate over 50 proprietary and open-weight LLMs on SORRY-Bench, analyzing their distinctive safety refusal behaviors. We hope our effort provides a building block for systematic evaluations of LLMs' safety refusal capabilities, in a balanced, granular, and efficient manner. Benchmark demo, data, code, and models are available through https://sorry-bench.github.io.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 55 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

43 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
26 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  10. 2024A StrongREJECT for Empty Jailbreakscited by 352
  11. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  13. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  14. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  15. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  16. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  17. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  18. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  19. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  20. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  21. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  22. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  23. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  24. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  25. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  26. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 91; 2026: 104.

224 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  5. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  6. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  9. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  10. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
  11. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  12. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
38 more
  1. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  2. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  3. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  4. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  5. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  6. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  7. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  9. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  10. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  11. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  12. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  13. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
  14. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 29
  15. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  16. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  17. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  18. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 25
  19. 2025o3-mini vs DeepSeek-R1: Which One is Safer?cited by 24
  20. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  21. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 23
  22. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  23. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  24. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  25. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 20
  26. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  27. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
  28. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  29. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  30. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 18
  31. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 17
  32. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 17
  33. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  34. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 15
  35. 2025Multi-Turn Jailbreaks Are Simpler Than They Seemcited by 15
  36. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
  37. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  38. 2024Superficial Safety Alignment Hypothesiscited by 14

50 most cited of 224 All 224 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models
  2. 2026Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
  3. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)
  4. 2025LLM-Safety Evaluations Lack Robustness
  5. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  6. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models
  7. 2026Ablating Safety: Mechanisms for Removing Alignment in Language Models for Security Applications
  8. 2026Understanding the Effects of Safety Unalignment on Large Language Models