mlsanityData to 14 Oct 2026

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

arXiv 2609.36254 ↗First posted 28 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning also fits: Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 81 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

58 of 81 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models4 Feb
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 54

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2024A Survey on Large Language Models for Code Generationcited by 658
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  12. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
38 more
  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  2. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  4. 2024Alignment faking in large language modelscited by 360
  5. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  7. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  8. 2025Qwen3Guard Technical Reportcited by 155
  9. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  10. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  11. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  12. 2025Stress Testing Deliberative Alignment for Anti-Scheming Trainingcited by 86
  13. 2025Safety in Large Reasoning Models: A Surveycited by 64
  14. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 59
  15. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
  16. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 45
  17. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 38
  18. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 32
  19. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 29
  20. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 28
  21. 2024Granite Guardiancited by 25
  22. 2026Reasoning Models Struggle to Control their Chains of Thoughtcited by 25
  23. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  24. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  25. 2025Reasoning as an Adaptive Defense for Safetycited by 23
  26. 2025DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarioscited by 19
  27. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 16
  28. 2025Chain-of-Thought Hijackingcited by 16
  29. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  30. 2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
  31. 2025Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RLcited by 10
  32. 2025ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Modelscited by 9
  33. 2026A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringcited by 8
  34. 2025Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Modelscited by 7
  35. 2025D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Modelscited by 7
  36. 2025OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulationcited by 5
  37. 2025SafeRBench: Dissecting the Reasoning Safety of Large Language Modelscited by 4
  38. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
  2. 2026DeShortcut-Align: Decoupling Spurious Shortcuts for Robust Safety Alignment in Large Reasoning Models
  3. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
  4. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Models
  5. 2026Internalizing Safety Understanding in Large Reasoning Models via Verification
  6. 2026Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models
  7. 2026Reasoning Structure Matters for Safety Alignment of Reasoning Models
  8. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations