# Rethinking On-Policy Self-Distillation for Thinking Models

arXiv: https://arxiv.org/abs/2607.05184 · HTML: https://mlsanity.com/paper/2607.05184/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 16 of 16)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Olmo 3](https://mlsanity.com/paper/2512.13961/) (2025)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability](https://mlsanity.com/paper/2411.19943/) (2024)
- [Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision](https://mlsanity.com/paper/2604.12002/) (2026)
- [POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration](https://mlsanity.com/paper/2601.18779/) (2026)
- [HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation](https://mlsanity.com/paper/2603.23871/) (2026)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [Forking Paths in Neural Text Generation](https://mlsanity.com/paper/2412.07961/) (2024)
- [Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes](https://mlsanity.com/paper/2601.18795/) (2026)

## Built on by (most cited first, 30 of 39)

- [Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09228/) (2026)
- [Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement](https://mlsanity.com/paper/2608.31046/) (2026)
- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning](https://mlsanity.com/paper/2608.01589/) (2026)
- [Rethinking Privileged Information in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.18271/) (2026)
- [Simple-OPD: Demystifying Warm-up for On-policy Distillation](https://mlsanity.com/paper/2608.06802/) (2026)
- [Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09263/) (2026)
- [Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL](https://mlsanity.com/paper/2608.14945/) (2026)
- [Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation](https://mlsanity.com/paper/2609.08798/) (2026)
- [What Does Privileged Information Add to On-Policy Self-Distillation?](https://mlsanity.com/paper/2609.20612/) (2026)
- [What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.25623/) (2026)
- [Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders](https://mlsanity.com/paper/2609.35210/) (2026)
- [Solving Without Stopping: On-Policy Distillation at Small Scale](https://mlsanity.com/paper/2609.37326/) (2026)
- [Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight](https://mlsanity.com/paper/2610.08077/) (2026)
- [A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching](https://mlsanity.com/paper/2610.10447/) (2026)
- [ProBel: Propaganda Detection with Techniques, Spans, and Explanations](https://mlsanity.com/paper/2608.22388/) (2026)
- [One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation](https://mlsanity.com/paper/2608.25936/) (2026)
- [VERPO: Verified Evidence Regularized Policy Optimization](https://mlsanity.com/paper/2609.06100/) (2026)
- [Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models](https://mlsanity.com/paper/2609.09957/) (2026)
- [PSD: Pseudo Self-Distillation of Memory Representation Capabilities for LLM Agents](https://mlsanity.com/paper/2609.23449/) (2026)
- [Instruct, Not Answer: Using Instruction Privileges in On-Policy Context Distillation](https://mlsanity.com/paper/2609.32201/) (2026)
- [Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents](https://mlsanity.com/paper/2609.33391/) (2026)
- [TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models](https://mlsanity.com/paper/2609.33414/) (2026)
- [DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation](https://mlsanity.com/paper/2609.33711/) (2026)
- [Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR](https://mlsanity.com/paper/2609.37825/) (2026)
- [Diagnosing On-Policy Self-Distillation for Reasoning Language Models](https://mlsanity.com/paper/2609.39118/) (2026)
- [Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents](https://mlsanity.com/paper/2609.39957/) (2026)
- [PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning](https://mlsanity.com/paper/2610.00949/) (2026)
- [The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2610.02191/) (2026)
- [E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation](https://mlsanity.com/paper/2610.05048/) (2026)

## Abstract (arXiv)

Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.