The Thinking Leakage in Reinforcement Learning for LLMs

arXiv 2610.14270 ↗First posted 9 Oct 2026cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning also fits: Reinforcement learning for reasoning

Abstract (arXiv)

Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens. A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance. We provide evidence that much of this gain can be explained by thinking leakage: the gain comes largely from partially reactivating the reasoning abilities of the original model's THINK mode instead of learning enhanced reasoning behavior. We train Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK mode. As accuracy improves, responses become longer and more reflective, are increasingly judged by the original model to resemble its own thinking, and the model's internal activations shift toward its THINK mode. We then investigate at the level of weights, activations, and attention heads: scaling the RL update further elicits THINK-mode behavior and approaches THINK-mode accuracy but does not exceed it; in Qwen3-8B, replacing a low-dimensional THINK-related activation subspace of the RL-trained model with its original-model value removes most of the RL gain, and the reverse exchange transfers a considerable fraction of the gain to the original model; and the RL-induced shift passes through the attention heads that read the empty think block. We further identify a possible mechanism in the GRPO training signal: responses favored by the advantage-weighted update are more THINK-like, and a gradient step computed from this signal moves the model toward its THINK state. Thus, gains in NO-THINK mode can largely reflect reactivation of reasoning abilities already present in the original model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 47 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

42 of 47 references.

Closest ancestors · 1

Same direction, newest first

2024Training Large Language Models to Reason in a Continuous Latent Space9 Dec

Other references · 41

2017Proximal Policy Optimization Algorithmscited by 15,215
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
2022Locating and Editing Factual Associations in GPTcited by 2,691
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
2022Editing Models with Task Arithmeticcited by 1,120
2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
2023Steering Language Models With Activation Engineeringcited by 922

29 more

2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
2021Causal Abstractions of Neural Networkscited by 422
2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
2023In-Context Learning Creates Task Vectorscited by 312
2023Function Vectors in Large Language Modelscited by 308
2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
2025Reasoning Models Can Be Effective Without Thinkingcited by 187
2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 147
2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
2025Thinkless: LLM Learns When to Thinkcited by 113
2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 66
2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
2025Base Models Know How to Reason, Thinking Models Learn Whencited by 27
2025Reasoning-Finetuning Repurposes Latent Representations in Base Modelscited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way