mlsanityData to 7 Oct 2026

SaRO: Enhancing LLM Safety through Reasoning-based Alignment

arXiv 2504.09420 ↗First posted 13 Apr 2025cs.CLcited by 26

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Current safety alignment techniques for large language models (LLMs) face two key challenges: (1) under-generalization, which leaves models vulnerable to novel jailbreak attacks, and (2) over-alignment, which leads to the excessive refusal of benign instructions. Our preliminary investigation reveals semantic overlap between jailbreak/harmful queries and normal prompts in embedding space, suggesting that more effective safety alignment requires a deeper semantic understanding. This motivates us to incorporate safety-policy-driven reasoning into the alignment process. To this end, we propose the Safety-oriented Reasoning Optimization Framework (SaRO), which consists of two stages: (1) Reasoning-style Warmup (RW) that enables LLMs to internalize long-chain reasoning through supervised fine-tuning, and (2) Safety-oriented Reasoning Process Optimization (SRPO) that promotes safety reflection via direct preference optimization (DPO). Extensive experiments demonstrate the superiority of SaRO over traditional alignment methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 40 of its 57 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

40 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities17 Feb
  2. 2024OpenAI o1 System Card21 Dec
  3. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
25 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  8. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  9. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  10. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  11. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  12. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  13. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  14. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  15. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  16. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  17. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  18. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  19. 2024Rule Based Rewards for Language Model Safetycited by 106
  20. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  21. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  22. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  23. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  24. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  25. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  2. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  4. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  5. 2025Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skillscited by 12
  6. 2025UnsafeChain: Enhancing Reasoning Model Safety via Hard Casescited by 7
  7. 2025Should LLM Safety Be More Than Refusing Harmful Instructions?cited by 6
  8. 2026Overview of PAN 2026: Voight-Kampff Generative AI Detection, Text Watermarking, Multi-Author Writing Style Analysis, Generative Plagiarism Detection, and Reasoning Trajectory Detectioncited by 3
  9. 2025Does More Inference-Time Compute Really Help Robustness?cited by 2
  10. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Abilitycited by 2
  11. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignmentcited by 2
  12. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
14 more
  1. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 1
  2. 2025AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teamingcited by 1
  3. 2026RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoningcited by 1
  4. 2026Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base modelcited by 1
  5. 2026Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectoriescited by 1
  6. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimizationcited by 1
  7. 2026ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agentscited by 1
  8. 2025SafeConstellations: Mitigating Over-Refusals in LLMs Through Task-Aware Representation Steeringcited by 0
  9. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Modelscited by 0
  10. 2026How Does Prefix Matter in Reasoning Model Tuning?cited by 0
  11. 2026The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMscited by 0
  12. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0
  13. 2026Are LLMs ready for HardChoices?cited by 0
  14. 2026Compliance2LoRA: Personalizable On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025STAIR: Improving Safety Alignment with Introspective Reasoning
  2. 2024Superficial Safety Alignment Hypothesis
  3. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning
  4. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety
  5. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  6. 2026Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning
  7. 2026MOSAIC: Composable Safety Alignment with Modular Control Tokens
  8. 2025RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards