# The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

arXiv: https://arxiv.org/abs/2605.11182 · HTML: https://mlsanity.com/paper/2605.11182/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models](https://mlsanity.com/paper/2604.08527/) (2026)

## Built on by (most cited first, 30 of 45)

- [RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation](https://mlsanity.com/paper/2606.11709/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning](https://mlsanity.com/paper/2608.01589/) (2026)
- [Rethinking Privileged Information in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.18271/) (2026)
- [MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation](https://mlsanity.com/paper/2605.27186/) (2026)
- [Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2606.12634/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation](https://mlsanity.com/paper/2607.10805/) (2026)
- [CriPO: Enhancing Rubric-based RL via Self-Distillation](https://mlsanity.com/paper/2607.18082/) (2026)
- [Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling](https://mlsanity.com/paper/2608.11829/) (2026)
- [Tail-Aware Top-k On-Policy Distillation](https://mlsanity.com/paper/2608.14728/) (2026)
- [RISE: Recursive Improvement via Self-Extrapolating Policy Distillation](https://mlsanity.com/paper/2609.05295/) (2026)
- [Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions](https://mlsanity.com/paper/2606.09076/) (2026)
- [Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding](https://mlsanity.com/paper/2606.18101/) (2026)
- [Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09263/) (2026)
- [The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation](https://mlsanity.com/paper/2607.24720/) (2026)
- [DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation](https://mlsanity.com/paper/2607.29078/) (2026)
- [When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation](https://mlsanity.com/paper/2608.27960/) (2026)
- [DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer](https://mlsanity.com/paper/2606.04694/) (2026)
- [Le Critique: Privileged Value Functions for LLM Reinforcement Learning](https://mlsanity.com/paper/2608.16739/) (2026)
- [Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders](https://mlsanity.com/paper/2609.35210/) (2026)
- [SR-OPSD: Self-Referenced On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09745/) (2026)
- [SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation](https://mlsanity.com/paper/2608.14647/) (2026)
- [Distillation as Probability Transport: Routed On-Policy Distillation](https://mlsanity.com/paper/2609.08337/) (2026)
- [Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training](https://mlsanity.com/paper/2609.31900/) (2026)
- [Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning](https://mlsanity.com/paper/2609.33200/) (2026)
- [DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation](https://mlsanity.com/paper/2609.33711/) (2026)
- [UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents](https://mlsanity.com/paper/2609.34036/) (2026)
- [Unbiased Top-k Estimation for On-Policy Distillation](https://mlsanity.com/paper/2609.34447/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.