mlsanityData to 7 Oct 2026

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

arXiv 2604.13016 ↗First posted 14 Apr 2026cs.LG cs.AI cs.CLcited by 262

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 47 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

43 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  2. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation11 Mar
  3. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillation5 Mar
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan
  5. 2025Small Models Struggle to Learn from Strong Reasoners17 Feb
  6. 2025Distillation Scaling Laws12 Feb

Other references · 37

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
25 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  3. 2016Sequence-Level Knowledge Distillationcited by 761
  4. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  5. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  6. 2019On the Efficacy of Knowledge Distillationcited by 377
  7. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  8. 2026Reinforcement Learning via Self-Distillationcited by 285
  9. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  10. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  11. 2026Self-Distillation Enables Continual Learningcited by 212
  12. 2026MiMo-V2-Flash Technical Reportcited by 163
  13. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  14. 2026On-Policy Context Distillation for Language Modelscited by 161
  15. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  16. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  17. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  18. 2026Self-Distilled RLVRcited by 109
  19. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  20. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 57
  21. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  22. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 39
  23. 2026How Far Can Unsupervised RLVR Scale LLM Training?cited by 26
  24. 2026Online Experiential Learning for Language Modelscited by 24
  25. 2026Self-Distillation for Multi-Token Predictioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 262.

262 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  2. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
  3. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  4. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  5. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  6. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 28
  7. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  8. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 23
  9. 2026On the Position Bias of On-Policy Distillationcited by 23
  10. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 21
  11. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  12. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
38 more
  1. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  2. 2026On the Geometry of On-Policy Distillationcited by 17
  3. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
  4. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  5. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  6. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 14
  7. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  8. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 14
  9. 2026Trajectory-Refined Distillationcited by 14
  10. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 14
  11. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulationscited by 14
  12. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  13. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Informationcited by 12
  14. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  15. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Samplingcited by 12
  16. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  17. 2026Rubric-based On-policy Distillationcited by 11
  18. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  19. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  20. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  21. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  22. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  23. 2026Crosslingual On-Policy Self-Distillation for Multilingual Reasoningcited by 10
  24. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  25. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoningcited by 10
  26. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  27. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  28. 2026On-policy Distillation with Verifiable Rewardcited by 10
  29. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 9
  30. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 9
  31. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 9
  32. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 9
  33. 2026OPRD: On-Policy Representation Distillationcited by 9
  34. 2026DOPD: Dual On-policy Distillationcited by 9
  35. 2026Visual Contrastive Self-Distillationcited by 9
  36. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  37. 2026OneReason Technical Reportcited by 8
  38. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8

50 most cited of 262 All 262 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
  3. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  4. 2026Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
  5. 2026A Survey of On-Policy Distillation for Large Language Models
  6. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  7. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
  8. 2026Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation