mlsanityData to 7 Oct 2026

OR-Bench: An Over-Refusal Benchmark for Large Language Models

arXiv 2405.20947 ↗First posted 31 May 2024cs.CL cs.AIcited by 222

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) require careful safety alignment to prevent malicious outputs. While significant research focuses on mitigating harmful content generation, the enhanced safety often come with the side effect of over-refusal, where LLMs may reject innocuous prompts and become less helpful. Although the issue of over-refusal has been empirically observed, a systematic measurement is challenging due to the difficulty of crafting prompts that can elicit the over-refusal behaviors of LLMs. This study proposes a novel method for automatically generating large-scale over-refusal datasets. Leveraging this technique, we introduce OR-Bench, the first large-scale over-refusal benchmark. OR-Bench comprises 80,000 over-refusal prompts across 10 common rejection categories, a subset of around 1,000 hard prompts that are challenging even for state-of-the-art LLMs, and an additional 600 toxic prompts to prevent indiscriminate responses. We then conduct a comprehensive study to measure the over-refusal of 32 popular LLMs across 8 model families. Our datasets are publicly available at https://huggingface.co/bench-llms and our codebase is open-sourced at https://github.com/justincui03/or-bench. We hope this benchmark can help the community develop better safety aligned models.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 67 of its 105 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

67 of 105 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
38 more
  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2024Mixtral of Expertscited by 1,785
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  13. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  15. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  16. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  17. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  18. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  19. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  20. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  21. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  22. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  23. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  24. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  25. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  26. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  27. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  28. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  29. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  30. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  31. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  32. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  33. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  34. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  35. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  36. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  37. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  38. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 80; 2026: 123.

222 papers.

Sort:

By area:

  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  2. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  3. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  6. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  7. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  8. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  9. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  10. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  11. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  12. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
38 more
  1. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  2. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  3. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  4. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  5. 2025Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUscited by 22
  6. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  7. 2025Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuningcited by 22
  8. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  9. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  10. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 17
  11. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
  12. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  13. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  14. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 15
  15. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  16. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  17. 2025RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguardscited by 14
  18. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  19. 2026YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Modelscited by 14
  20. 2024You Know What I'm Saying: Jailbreak Attack via Implicit Referencecited by 13
  21. 2025Consistency Training Helps Stop Sycophancy and Jailbreakscited by 13
  22. 2024Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilitiescited by 12
  23. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  24. 2025A Pragmatic View of AI Personhoodcited by 12
  25. 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectoriescited by 12
  26. 2025Difficulties with Evaluating a Deception Detector for AIscited by 11
  27. 2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
  28. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  29. 2025Self-Destructive Language Modelcited by 10
  30. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  31. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
  32. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
  33. 2026Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenderscited by 10
  34. 2025FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Taskscited by 9
  35. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  36. 2025A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Modelscited by 9
  37. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agentscited by 9
  38. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8

50 most cited of 222 All 222 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?
  2. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  3. 2026Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding
  4. 2025Refusal Direction is Universal Across Safety-Aligned Languages
  5. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
  6. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models
  7. 2024Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment
  8. 2026How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense