mlsanityData to 14 Oct 2026

Do Thinking Tokens Help with Safety?

arXiv 2606.25013 ↗First posted 23 Jun 2026cs.LG cs.AI cs.CLcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also generally believed that this more "deliberative" mode should improve alignment and safety, by providing the model a safe space to consider whether its planned answer to a request violates its safety principles. We present evidence that this intuition is not always correct. Across frontier open-weight reasoning models spanning GPT-OSS, Qwen, Olmo, and Phi families, we find that the eventual refusal/compliance outcome is already strongly predictable via a trained head on the first token's hidden representation (0.84-0.95 AUROC and ~88% balanced accuracy for predicting refusal/compliance) before any visible thinking. The thinking process turns out to be more akin to prefix completion than to deliberative revision, with the final outcome rarely changing after the first ~20% of thinking, despite giving the appearance of deliberation at the text level (~74% of text-level deliberations occur when the response distribution is already locked to one refusal/compliance side). We also find that existing inference-time and training-based safety interventions, despite being motivated by the goal of inducing deliberation, largely shift model behavior toward over-refusal while suppressing already-scarce deliberation signals. Our results suggest that safety behavior in current reasoning models is much less deliberative than commonly assumed, and highlight the need for methods that induce real safety deliberation.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 53 of its 67 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

53 of 67 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoning4 Feb
  2. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Models30 Jan
  3. 2026How Does the Thinking Step Influence Model Safety? An Entropy-based Safety Reminder for LRMs7 Jan
  4. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?7 Oct
  5. 2025R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge1 Aug
  6. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning22 May
  7. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment20 May
  8. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning12 May
  9. 2025Safety in Large Reasoning Models: A Survey24 Apr
  10. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Data2 Apr
  11. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R118 Feb
  12. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities17 Feb
1 more
  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec

Other references · 40

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  12. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
28 more
  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  4. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  5. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  6. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  7. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  8. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  9. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 170
  10. 2025Qwen3Guard Technical Reportcited by 155
  11. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  12. 2025Phi-4-reasoning Technical Reportcited by 151
  13. 2025Olmo 3cited by 129
  14. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  15. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  16. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  17. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 88
  18. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  19. 2024Best-of-N Jailbreakingcited by 76
  20. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 40
  21. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  22. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  23. 2025Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructionscited by 11
  24. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  25. 2025ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusalcited by 6
  26. 2026Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answerscited by 6
  27. 2025Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Followingcited by 3
  28. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Cost of Reasoning in non-English Languages: A Case Study on Japanesecited by 1
  2. 2026Many-body Tipping Dynamics of ChatGPT-like AIscited by 0
  3. 2026Can escalation channels redirect reward hacking toward defect disclosure?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026First Token Matters: Understanding Safety Collapse in Large Reasoning Models
  2. 2025Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Steps
  3. 2026Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering
  4. 2025SafeRBench: Dissecting the Reasoning Safety of Large Language Models
  5. 2026Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model
  6. 2025Where Do Reasoning Models Refuse?
  7. 2024Rethinking Thinking Tokens: Understanding Why They Underperform in Practice
  8. 2025Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought