mlsanityData to 7 Oct 2026

A Formula-Driven Survey and Research Agenda for On-Policy Distillation

arXiv 2606.22793 ↗First posted 22 Jun 2026cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) trains an LLM on states induced by the current or recent student policy: the student generates complete or partial rollouts, a teacher or self-teacher scores the resulting tokens under their generated contexts, and dense log-probability, logit, or distributional signals are converted into post-training updates. This survey studies OPD as a feedback-to-update problem rather than a single loss family. We develop a formula-driven taxonomy from two routes -- direct distributional losses and policy-gradient-style log-ratio updates -- and use it to organize core methods, verifier- or outcome-guided hybrids, industrial reports, framework implementations, failure modes, and stabilization recipes under explicit evidence boundaries. The taxonomy shows that OPD effectiveness depends not only on KL direction or teacher access, but also on state compatibility, support construction, temporal credit, vocabulary-level probability routing, gates and weights, and regularization. We further separate two mechanisms often conflated in sampled-token OPD stability discussions. Temporal credit asks how teacher-student log-ratio returns should weight sampled actions across a rollout; vocabulary routing asks where probability mass should move when negative feedback suppresses a sampled token. This distinction yields bias boundaries for immediate, return-to-go, discounted, and baseline-corrected estimators, motivates GAE-OPD as a value-based hypothesis for log-ratio returns, and motivates Counterfactual Routed OPD (CR-OPD) for routing probability mass toward teacher-supported, student-reachable alternatives. We close by mapping actionability diagnostics, failure mechanisms, case studies, open problems, and a reporting checklist onto the same feedback-to-update variables.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 82 of its 93 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

82 of 93 references.

References · 82

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2016Sequence-Level Knowledge Distillationcited by 761
38 more
  1. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  2. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  3. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  4. 2026On-Policy Context Distillation for Language Modelscited by 161
  5. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  6. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  7. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  8. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  9. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  10. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  11. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  12. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  13. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  14. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  15. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  16. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  17. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
  18. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  19. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43
  20. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  21. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 39
  22. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  23. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 38
  24. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
  25. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  26. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learningcited by 35
  27. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 32
  28. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  29. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 30
  30. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  31. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  32. 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillationcited by 26
  33. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  34. 2026Online Experiential Learning for Language Modelscited by 24
  35. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  36. 2025ToDi: Token-wise Distillation via Fine-Grained Divergence Controlcited by 23
  37. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 23
  38. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 21

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026SAF-OPD: Stable Advantage Fusion for On-Policy Distillationcited by 4
  2. 2026DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillationcited by 1
  3. 2026One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillationcited by 0
  4. 2026Dual-Vocabulary Language Model for Cross-Tokenizer Distillationcited by 0
  5. 2026Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Modelscited by 0