Diagnosing On-Policy Self-Distillation for Reasoning Language Models
On the map: LLM reasoning → Prompting and chain-of-thought reasoning also fits: Test-time compute and efficient thinking
Abstract (arXiv)
On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 41 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
40 of 41 references.
Closest ancestors · 9
Same direction, newest first
- 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation30 Jul
- 2026Rethinking On-Policy Self-Distillation for Thinking Models6 Jul
- 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think2 Jul
- 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation10 Jun
- 2026Self-Supervised On-Policy Distillation for Reasoning Language Models17 May
- 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning8 May
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
- 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan
Other references · 31
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2017Decoupled Weight Decay Regularizationcited by 1,722
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
19 more
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
- 2025Group Sequence Policy Optimizationcited by 788
- 2024DeepSeek-V3 Technical Reportcited by 724
- 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
- 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
- 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
- 2026Self-Distilled RLVRcited by 109
- 2026Kimi K3: Open Frontier Intelligencecited by 99
- 2025Olmo 3cited by 95
- 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
- 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43
- 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
- 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 24
- 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
- 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulationscited by 14
- 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 12
- 2026Rethinking Privileged Information in On-Policy Self-Distillationcited by 6
- 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 4
- 2026Less is More: Early Stopping Rollout for On-Policy Distillationcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
- 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
- 2026Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning
- 2026Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation
- 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
- 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
- 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains
- 2026SOD: Step-wise On-policy Distillation for Small Language Model Agents