# THINKSAFE: Self-Generated Safety Alignment for Reasoning Models

arXiv: https://arxiv.org/abs/2601.23143 · HTML: https://mlsanity.com/paper/2601.23143/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 10 of its 43 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable](https://mlsanity.com/paper/2503.00555/) (2025)
- [Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning](https://mlsanity.com/paper/2502.09673/) (2025)
- [LoRA is All You Need for Safety Alignment of Reasoning LLMs](https://mlsanity.com/paper/2507.17075/) (2025)
- [Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success](https://mlsanity.com/paper/2601.18175/) (2026)

## Built on by (most cited first, 10 of 10)

- [Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation](https://mlsanity.com/paper/2605.15239/) (2026)
- [Do Thinking Tokens Help with Safety?](https://mlsanity.com/paper/2606.25013/) (2026)
- [Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models](https://mlsanity.com/paper/2605.25443/) (2026)
- [It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs](https://mlsanity.com/paper/2605.20258/) (2026)
- [SHARD: Safe and Helpful Alignment via Self-Reframing Distillation](https://mlsanity.com/paper/2606.15517/) (2026)
- [Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation](https://mlsanity.com/paper/2607.14256/) (2026)
- [Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs](https://mlsanity.com/paper/2608.09542/) (2026)
- [Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal](https://mlsanity.com/paper/2609.04482/) (2026)
- [EOPSA: Efficient On-Policy Self-Distilled Safety Alignment](https://mlsanity.com/paper/2609.34519/) (2026)
- [Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge](https://mlsanity.com/paper/2610.07532/) (2026)

## Abstract (arXiv)

Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We formalize safety realignment as a KL projection onto the safe simplex and prove that the student's own safety-filtered distribution is the unique KL-optimal target, while any external teacher incurs an irreducible excess KL penalty. Guided by this analysis, we propose ThinkSafe, a self-generated alignment framework that restores safety without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, which preserves the KL-optimal target while increasing the acceptance rate. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency, and achieves superior safety and comparable reasoning to GRPO with roughly an order of magnitude less compute. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe and https://huggingface.co/Seanie-lee/collections.