# RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability

arXiv: https://arxiv.org/abs/2504.10081 · HTML: https://mlsanity.com/paper/2504.10081/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [AlphaMath Almost Zero: Process Supervision without Process](https://mlsanity.com/paper/2405.03553/) (2024)
- [SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities](https://mlsanity.com/paper/2502.12025/) (2025)
- [Simulating Classroom Education with LLM-Empowered Agents](https://mlsanity.com/paper/2406.19226/) (2024)
- [Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable](https://mlsanity.com/paper/2503.00555/) (2025)
- [The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1](https://mlsanity.com/paper/2502.12659/) (2025)
- [STAIR: Improving Safety Alignment with Introspective Reasoning](https://mlsanity.com/paper/2502.02384/) (2025)
- [Step-level Value Preference Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2406.10858/) (2024)
- [STAR-1: Safer Alignment of Reasoning LLMs with 1K Data](https://mlsanity.com/paper/2504.01903/) (2025)
- [Enhancing LLM Safety via Constrained Direct Preference Optimization](https://mlsanity.com/paper/2403.02475/) (2024)
- [CodeMind: Evaluating Large Language Models for Code Reasoning](https://mlsanity.com/paper/2402.09664/) (2024)
- [Safety Evaluation of DeepSeek Models in Chinese Contexts](https://mlsanity.com/paper/2502.11137/) (2025)

## Built on by (most cited first, 30 of 59)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [Mitigating Overthinking in Large Reasoning Models via Manifold Steering](https://mlsanity.com/paper/2505.22411/) (2025)
- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment](https://mlsanity.com/paper/2505.14667/) (2025)
- [Reasoning as an Adaptive Defense for Safety](https://mlsanity.com/paper/2507.00971/) (2025)
- [SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning](https://mlsanity.com/paper/2505.16186/) (2025)
- [A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law](https://mlsanity.com/paper/2505.02665/) (2025)
- [Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models](https://mlsanity.com/paper/2509.01909/) (2025)
- [DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios](https://mlsanity.com/paper/2510.15501/) (2025)
- [How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2505.15404/) (2025)
- [Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space](https://mlsanity.com/paper/2505.21277/) (2025)
- [Large Reasoning Models Learn Better Alignment from Flawed Thinking](https://mlsanity.com/paper/2510.00938/) (2025)
- [SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization](https://mlsanity.com/paper/2511.12982/) (2025)
- [Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions](https://mlsanity.com/paper/2509.01444/) (2025)
- [Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention](https://mlsanity.com/paper/2509.24393/) (2025)
- [Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?](https://mlsanity.com/paper/2510.06036/) (2025)
- [ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning](https://mlsanity.com/paper/2507.11500/) (2025)
- [Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Models](https://mlsanity.com/paper/2505.19690/) (2025)
- [UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases](https://mlsanity.com/paper/2507.21652/) (2025)
- [What Matters For Safety Alignment?](https://mlsanity.com/paper/2601.03868/) (2026)
- [R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge](https://mlsanity.com/paper/2508.00324/) (2025)
- [Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check](https://mlsanity.com/paper/2509.11629/) (2025)
- [When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models](https://mlsanity.com/paper/2510.21285/) (2025)
- [Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?](https://mlsanity.com/paper/2510.06410/) (2025)
- [Strong but Brittle: Simple Attacks Subvert Reasoning-based Safety Guardrails](https://mlsanity.com/paper/2510.11570/) (2025)
- [ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments](https://mlsanity.com/paper/2508.04204/) (2025)
- [Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation](https://mlsanity.com/paper/2508.15370/) (2025)
- [Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation](https://mlsanity.com/paper/2509.14760/) (2025)

## Abstract (arXiv)

Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have been rapidly progressing and achieving breakthrough performance on complex reasoning tasks such as mathematics and coding. However, the open-source R1 models have raised safety concerns in wide applications, such as the tendency to comply with malicious queries, which greatly impacts the utility of these powerful models in their applications. In this paper, we introduce RealSafe-R1 as safety-aligned versions of DeepSeek-R1 distilled models. To train these models, we construct a dataset of 15k safety-aware reasoning trajectories generated by DeepSeek-R1, under explicit instructions for expected refusal behavior. Both quantitative experiments and qualitative case studies demonstrate the models' improvements, which are shown in their safety guardrails against both harmful queries and jailbreak attacks. Importantly, unlike prior safety alignment efforts that often compromise reasoning performance, our method preserves the models' reasoning capabilities by maintaining the training data within the original distribution of generation. Model weights of RealSafe-R1 are open-source at https://huggingface.co/RealSafe.