mlsanityData to 7 Oct 2026

A Survey of On-Policy Distillation for Large Language Models

arXiv 2604.00626 ↗First posted 1 Apr 2026cs.LG cs.CLcited by 148

Not on the map. Why?

Cited by 39 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As Large Language Models continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become an important engineering problem, and knowledge distillation remains a common technique for this transfer. The prevailing recipe in industrial pipelines, static imitation of teacher-generated text, carries a structural weakness that grows more severe as tasks become longer and more reasoning-intensive. Because the student is trained on flawless teacher prefixes but generates its own at inference, small errors tend to accumulate into trajectories it has rarely been trained to recover from, and the resulting exposure bias has been shown to scale roughly with the square of sequence length. On-Policy Distillation reorganizes the training loop around this observation by having the teacher provide feedback on what the student actually produces, with the goal of reducing the compounding term toward linear and reframing distillation as an iterative correction process rather than single-pass imitation. The resulting literature has expanded along divergence design, reward-guided optimization, and self-play, yet contributions remain scattered across the knowledge distillation, RLHF, and imitation learning communities without a unified treatment. This survey provides such a treatment. We formalize OPD as f-divergence minimization over student-sampled trajectories, organize the field along three design axes (what to optimize, where the signal comes from, and how to stabilize training in practice), and consolidate success conditions, recurring failure modes, and the connection between OPD and KL-constrained reinforcement learning. We close with open problems that emerge from this synthesis, including distillation scaling laws, uncertainty-aware feedback, agent-level distillation, and the growing overlap between knowledge distillation and RL.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Apr 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 148.

148 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  2. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  3. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  4. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  5. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
  6. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  7. 2026On the Position Bias of On-Policy Distillationcited by 23
  8. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  9. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
  10. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 17
  11. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  12. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
38 more
  1. 2026Trajectory-Refined Distillationcited by 14
  2. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  3. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  4. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  5. 2026Rubric-based On-policy Distillationcited by 11
  6. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  7. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  8. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  9. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  10. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  11. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 9
  12. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 9
  13. 2026DOPD: Dual On-policy Distillationcited by 9
  14. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  15. 2026On-Policy Delta Distillationcited by 8
  16. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  17. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillationcited by 7
  18. 2026PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformationcited by 7
  19. 2026Weak-to-Strong On-Policy Distillationcited by 7
  20. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  21. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  22. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
  23. 2026Latent On-Policy Self-Distillationcited by 6
  24. 2026Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoningcited by 5
  25. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  26. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 5
  27. 2026Breaking the Tokenizer Barrier: On-Policy Distillation across Model Familiescited by 5
  28. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  29. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scalingcited by 5
  30. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  31. 2026Healthcare AI GYM for Medical Agentscited by 4
  32. 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Controlcited by 4
  33. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoningcited by 4
  34. 2026CoMAP: Co-Evolving World Models and Agent Policies for LLM Agentscited by 4
  35. 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 4
  36. 2026MaineCoon: Pursuing A Real-Time Audio-Visual Social World Modelcited by 4
  37. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  38. 2026SAF-OPD: Stable Advantage Fusion for On-Policy Distillationcited by 4

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
  2. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  3. 2025Knowledge Distillation with Training Wheels
  4. 2026On-Policy Context Distillation for Language Models
  5. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  6. 2025UNDO: Understanding Distillation as Optimization
  7. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  8. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence