# Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable

arXiv: https://arxiv.org/abs/2503.00555 · HTML: https://mlsanity.com/paper/2503.00555/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning](https://mlsanity.com/paper/2501.12570/) (2025)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [CoT-Valve: Length-Compressible Chain-of-Thought Tuning](https://mlsanity.com/paper/2502.09601/) (2025)
- [SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities](https://mlsanity.com/paper/2502.12025/) (2025)
- [The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1](https://mlsanity.com/paper/2502.12659/) (2025)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment](https://mlsanity.com/paper/2310.00212/) (2023)
- [Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging](https://mlsanity.com/paper/2502.06876/) (2025)
- [To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models](https://mlsanity.com/paper/2502.12202/) (2025)
- [Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning](https://mlsanity.com/paper/2502.09673/) (2025)

## Built on by (most cited first, 30 of 122)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Effectively Controlling Reasoning Models through Thinking Intervention](https://mlsanity.com/paper/2503.24370/) (2025)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability](https://mlsanity.com/paper/2504.10081/) (2025)
- [GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning](https://mlsanity.com/paper/2505.11049/) (2025)
- [Assessing Judging Bias in Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2504.09946/) (2025)
- [100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models](https://mlsanity.com/paper/2505.00551/) (2025)
- [SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models](https://mlsanity.com/paper/2504.08813/) (2025)
- [Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers](https://mlsanity.com/paper/2506.15674/) (2025)
- [When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoning](https://mlsanity.com/paper/2505.15400/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment](https://mlsanity.com/paper/2505.14667/) (2025)
- [Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities](https://mlsanity.com/paper/2503.17979/) (2025)
- [Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets](https://mlsanity.com/paper/2506.05346/) (2025)
- [Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models](https://mlsanity.com/paper/2509.01909/) (2025)
- [How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2505.15404/) (2025)
- [Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2507.09662/) (2025)
- [Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills](https://mlsanity.com/paper/2506.12963/) (2025)
- [Can AI Perceive Physical Danger and Intervene?](https://mlsanity.com/paper/2509.21651/) (2025)
- [Large Reasoning Models Learn Better Alignment from Flawed Thinking](https://mlsanity.com/paper/2510.00938/) (2025)
- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2509.21882/) (2025)
- [DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models](https://mlsanity.com/paper/2509.24296/) (2025)
- [SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization](https://mlsanity.com/paper/2511.12982/) (2025)
- [Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention](https://mlsanity.com/paper/2509.24393/) (2025)
- [Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization](https://mlsanity.com/paper/2512.11391/) (2025)
- [Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation](https://mlsanity.com/paper/2605.15239/) (2026)
- [THINKSAFE: Self-Generated Safety Alignment for Reasoning Models](https://mlsanity.com/paper/2601.23143/) (2026)
- [ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models](https://mlsanity.com/paper/2505.17244/) (2025)
- [SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents](https://mlsanity.com/paper/2510.17017/) (2025)

## Abstract (arXiv)

Safety alignment is an important procedure before the official deployment of a Large Language Model (LLM). While safety alignment has been extensively studied for LLM, there is still a large research gap for Large Reasoning Models (LRMs) that equip with improved reasoning capability. We in this paper systematically examine a simplified pipeline for producing safety aligned LRMs. With our evaluation of various LRMs, we deliver two main findings: i) Safety alignment can be done upon the LRM to restore its safety capability. ii) Safety alignment leads to a degradation of the reasoning capability of LRMs. The two findings show that there exists a trade-off between reasoning and safety capability with the sequential LRM production pipeline. The discovered trade-off, which we name Safety Tax, should shed light on future endeavors of safety research on LRMs. As a by-product, we curate a dataset called DirectRefusal, which might serve as an alternative dataset for safety alignment. Our source code is available at https://github.com/git-disl/Safety-Tax.