# AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models

arXiv: https://arxiv.org/abs/2509.24269 · HTML: https://mlsanity.com/paper/2509.24269/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 28 of 28)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities](https://mlsanity.com/paper/2502.12025/) (2025)
- [H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking](https://mlsanity.com/paper/2502.12893/) (2025)
- [The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1](https://mlsanity.com/paper/2502.12659/) (2025)
- [STAR-1: Safer Alignment of Reasoning LLMs with 1K Data](https://mlsanity.com/paper/2504.01903/) (2025)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability](https://mlsanity.com/paper/2504.10081/) (2025)
- [Logical Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2502.09100/) (2025)
- [Advancing Reasoning in Large Language Models: Promising Methods and Approaches](https://mlsanity.com/paper/2502.03671/) (2025)
- [o3-mini vs DeepSeek-R1: Which One is Safer?](https://mlsanity.com/paper/2501.18438/) (2025)
- [To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models](https://mlsanity.com/paper/2502.12202/) (2025)
- [SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning](https://mlsanity.com/paper/2505.16186/) (2025)
- [How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2505.15404/) (2025)
- [The dark deep side of DeepSeek: Fine-tuning attacks against the safety alignment of CoT-enabled models](https://mlsanity.com/paper/2502.01225/) (2025)
- [Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Models](https://mlsanity.com/paper/2505.19690/) (2025)
- [UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases](https://mlsanity.com/paper/2507.21652/) (2025)

## Built on by (most cited first, 4 of 4)

- [Asynchronous Reasoning: Training-Free Interactive Thinking LLMs](https://mlsanity.com/paper/2512.10931/) (2025)
- [Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations](https://mlsanity.com/paper/2603.03332/) (2026)
- [Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories](https://mlsanity.com/paper/2605.08936/) (2026)
- [Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure](https://mlsanity.com/paper/2609.27763/) (2026)

## Abstract (arXiv)

Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the multi-step nature of CoT introduces new safety challenges that extend beyond conventional language model alignment. We identify a failure mode in current safety CoT tuning methods: the textitsnowball effect, where minor reasoning deviations progressively amplify throughout the thought process, leading to either harmful compliance or excessive refusal. This effect stems from models being trained to imitate perfect reasoning scripts without learning to self-correct. To address this limitation, we propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning. Our method involves constructing a dataset containing Temptation-Correction and Hesitation-Correction samples, where models learn to recover from harmful reasoning drifts and unnecessary cautions. Extensive experiments show that AdvChain significantly enhances robustness against jailbreak attacks and CoT hijacking while substantially reducing over-refusal on benign prompts, achieving a superior safety-utility balance without compromising reasoning capabilities. Our work establishes a new direction for building more robust and reliable reasoning models.