mlsanityData to 14 Oct 2026

Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation

arXiv 2607.10805 ↗First posted 12 Jul 2026cs.CL cs.LGcited by 5

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 2 more
Abstract (arXiv)

On-Policy Self-Distillation (OPSD) has emerged as a crucial paradigm for enhancing and aligning Large Language Models (LLMs). However, in complex reasoning tasks, OPSD paradoxically degrades downstream performance. In this paper, we systematically investigate this pathology and identify a severe optimization trap we define as Thinking Collapse -- a sharp decline in the model's native intermediate reasoning behavior, measured by epistemic-token density (ET per 1k). Through entropy-based gradient masking and token-level target analysis, we show that this collapse is triggered by aggressive teacher gradients at high-student-entropy decision forks, where student epistemic tokens are frequently suppressed into teacher non-epistemic targets and are highly concentrated in high pointwise student-teacher divergence regions. To resolve this optimization pathology, we propose Adaptive Dual-Perspective OPSD (AD-OPSD), a robust control framework that dynamically moderates the self-distillation objective. AD-OPSD selectively anchors high-suppression-risk sandboxed tokens to a reference prior derived from the frozen base model via an asymmetrical pointwise divergence gate, preserving native thinking capacity while retaining OPSD's error-correcting power. Extensive experiments across competitive mathematical benchmarks show that AD-OPSD improves over standard OPSD by up to +4.1% absolute average accuracy across diverse model scales and datasets. Further analysis demonstrates that AD-OPSD mitigates thinking collapse and generalizes robustly to different post-training paradigms.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 36 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

24 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think2 Jul
  2. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information12 May
  3. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR11 May
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  5. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  6. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  6. 2026Reinforcement Learning via Self-Distillationcited by 332
  7. 2026Self-Distillation Enables Continual Learningcited by 247
  8. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  9. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  10. 2026Self-Distilled RLVRcited by 131
  11. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  12. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
6 more
  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
  2. 2025Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubtcited by 18
  3. 2026Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoningcited by 5
  4. 2026Less is More: Early Stopping Rollout for On-Policy Distillationcited by 4
  5. 2026Better, Faster: Harnessing Self-Improvement in Large Reasoning Modelscited by 3
  6. 2025Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillationcited by 7
  2. 2026Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillationcited by 4
  3. 2026VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillationcited by 0
  4. 2026iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMscited by 0
  5. 2026The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  2. 2026Rethinking On-Policy Self-Distillation for Thinking Models
  3. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  4. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning
  5. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  6. 2026E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
  7. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
  8. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance