# FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning

arXiv: https://arxiv.org/abs/2505.08054 · HTML: https://mlsanity.com/paper/2505.08054/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 46 of its 65 references that are in the map.
- First posted on arXiv 12 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 46)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)

## Built on by (most cited first, 30 of 58)

- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Models](https://mlsanity.com/paper/2505.10846/) (2025)
- [Adversarial D'ej`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks](https://mlsanity.com/paper/2510.21910/) (2025)
- [Read the Scene, Not the Script: Outcome-Aware Safety for LLMs](https://mlsanity.com/paper/2510.04320/) (2025)
- [The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs](https://mlsanity.com/paper/2510.09905/) (2025)
- [When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAG](https://mlsanity.com/paper/2603.03919/) (2026)
- [EquiMem: Calibrating Shared Memory in Multi-Agent Debate via Game-Theoretic Equilibrium](https://mlsanity.com/paper/2605.09278/) (2026)
- [Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision](https://mlsanity.com/paper/2606.32038/) (2026)
- [Probing Association Biases in LLM Moderation Over-Sensitivity](https://mlsanity.com/paper/2505.23914/) (2025)
- [Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective](https://mlsanity.com/paper/2506.19028/) (2025)
- [Evaluating Large Language Models for Detecting Antisemitism](https://mlsanity.com/paper/2509.18293/) (2025)
- [Intent Laundering: AI Safety Datasets Are Not What They Seem](https://mlsanity.com/paper/2602.16729/) (2026)
- [Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinement](https://mlsanity.com/paper/2603.03323/) (2026)
- [STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systems](https://mlsanity.com/paper/2604.10286/) (2026)
- [Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations](https://mlsanity.com/paper/2604.27093/) (2026)
- [Do Thinking Tokens Help with Safety?](https://mlsanity.com/paper/2606.25013/) (2026)
- [How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries](https://mlsanity.com/paper/2512.01037/) (2025)
- [Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure](https://mlsanity.com/paper/2603.16734/) (2026)
- [Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs](https://mlsanity.com/paper/2603.27518/) (2026)
- [Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rules](https://mlsanity.com/paper/2604.06233/) (2026)
- [OPIUM: Mitigating Steering Externalities and Over-Refusal via Dual Objective Latent Optimization](https://mlsanity.com/paper/2607.19806/) (2026)
- [Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results](https://mlsanity.com/paper/2607.20090/) (2026)
- [Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences](https://mlsanity.com/paper/2506.00195/) (2025)
- [Understanding and Mitigating Over-refusal for Large Language Models via Representation Intervention](https://mlsanity.com/paper/2511.19009/) (2025)
- [C-DeltaTheta: Circuit-Restricted Weight Arithmetic for Selective Refusal](https://mlsanity.com/paper/2602.04521/) (2026)
- [SciIntBench: Measuring LLM Compliance with Research Integrity Norms Under Adversarial Framing](https://mlsanity.com/paper/2605.29468/) (2026)
- [SyPS: Measuring Sycophancy Prompt Sensitivity in Large Language Models](https://mlsanity.com/paper/2608.23837/) (2026)
- [Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration](https://mlsanity.com/paper/2609.25049/) (2026)
- [SafeConstellations: Mitigating Over-Refusals in LLMs Through Task-Aware Representation Steering](https://mlsanity.com/paper/2508.11290/) (2025)
- [Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs](https://mlsanity.com/paper/2509.05367/) (2025)

## Abstract (arXiv)

Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities.