mlsanityData to 14 Oct 2026

Large Reasoning Models Learn Better Alignment from Flawed Thinking

arXiv 2510.00938 ↗First posted 1 Oct 2025cs.LGcited by 13

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large reasoning models (LRMs) "think" by generating structured chain-of-thought (CoT) before producing a final answer, yet they still lack the ability to reason critically about safety alignment and are easily biased when a flawed premise is injected into their thought process. We propose RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a principled reinforcement learning (RL) method for post-training that explicitly teaches models to override flawed reasoning trajectories and reroute to safe and helpful responses. RECAP trains on a mixture of synthetically generated counter-aligned CoT prefills and standard prompts, requires no additional training cost or modifications beyond vanilla reinforcement learning from human feedback (RLHF), and substantially improves safety and jailbreak robustness, reduces overrefusal, and preserves core reasoning capability -- all while maintaining inference token budget. Extensive analysis shows that RECAP-trained models engage in self-reflection more frequently and remain robust under adaptive attacks, preserving safety even after repeated attempts to override their reasoning.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 38 of its 51 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

38 of 51 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning22 May
  2. 2025How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study21 May
  3. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment20 May
  4. 2025Safety in Large Reasoning Models: A Survey24 Apr
  5. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability14 Apr
  6. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignment13 Apr
  7. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Data2 Apr
  8. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable1 Mar
  9. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R118 Feb
  10. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities17 Feb
  11. 2025STAIR: Improving Safety Alignment with Introspective Reasoning4 Feb
  12. 2025Early External Safety Testing of OpenAI's o3-mini: Insights from the Pre-Deployment Evaluation29 Jan

Other references · 26

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2024A Survey on Large Language Models for Code Generationcited by 658
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  10. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  11. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  12. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
14 more
  1. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  2. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
  5. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  6. 2023Robust Principles: Architectural Design Principles for Adversarially Robust CNNscited by 46
  7. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  8. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 39
  9. 2024Backtracking Improves Generation Safetycited by 32
  10. 2024Granite Guardiancited by 25
  11. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  12. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  13. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  14. 2025Discovering Forbidden Topics in Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
  2. 2026Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interactioncited by 9
  3. 2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rulescited by 3
  4. 2026Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Modelscited by 3
  5. 2026Internalizing Safety Understanding in Large Reasoning Models via Verificationcited by 3
  6. 2025Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?cited by 2
  7. 2026Self-Guard: Defending Large Reasoning Models via enhanced self-reflectioncited by 1
  8. 2026Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectoriescited by 1
  9. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0
  10. 2026First Token Matters: Understanding Safety Collapse in Large Reasoning Modelscited by 0
  11. 2026Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposurecited by 0
  12. 2026Towards Mitigating Deceptive Safety Alignment in Large Reasoning Modelscited by 0
1 more
  1. 2026ReSI: Recursive Safety Improvement toward Resistant and Resilient AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
  2. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Models
  3. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away
  4. 2025SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learning
  5. 2025AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
  6. 2026Reasoning Structure Matters for Safety Alignment of Reasoning Models
  7. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment
  8. 2025Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models