mlsanityData to 7 Oct 2026

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

arXiv 2603.25562 ↗First posted 26 Mar 2026cs.LG cs.AI cs.CLcited by 122

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

32 of 42 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
20 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2016Sequence-Level Knowledge Distillationcited by 761
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  4. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  6. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  7. 2026Reinforcement Learning via Self-Distillationcited by 285
  8. 2026MiMo-V2-Flash Technical Reportcited by 163
  9. 2026On-Policy Context Distillation for Language Modelscited by 161
  10. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  11. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  12. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  13. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  14. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  15. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  16. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  17. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  18. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Modelscited by 9
  19. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  20. 2026Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RLcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 122.

122 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Self-Distilled RLVRcited by 109
  3. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  4. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  5. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  6. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  7. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  8. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 28
  9. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  10. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  11. 2026On the Geometry of On-Policy Distillationcited by 17
  12. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
38 more
  1. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 16
  2. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  3. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  4. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 14
  5. 2026Trajectory-Refined Distillationcited by 14
  6. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  7. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  8. 2026Rubric-based On-policy Distillationcited by 11
  9. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  10. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  11. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  12. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  13. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  14. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoningcited by 10
  15. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  16. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  17. 2026On-policy Distillation with Verifiable Rewardcited by 10
  18. 2026OPRD: On-Policy Representation Distillationcited by 9
  19. 2026Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillationcited by 9
  20. 2026DOPD: Dual On-policy Distillationcited by 9
  21. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidancecited by 8
  22. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillationcited by 8
  23. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvementcited by 8
  24. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  25. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  26. 2026PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformationcited by 7
  27. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rolloutscited by 7
  28. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
  29. 2026Self-Distilled Policy Gradientcited by 6
  30. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  31. 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agentscited by 6
  32. 2026Latent On-Policy Self-Distillationcited by 6
  33. 2026Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoningcited by 5
  34. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  35. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 5
  36. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  37. 2026Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillationcited by 5
  38. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scalingcited by 5

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
  3. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
  4. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  5. 2026Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
  6. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreement
  7. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
  8. 2026Reward-Gated On-Policy Distillation