mlsanityData to 7 Oct 2026

Entropy-Aware On-Policy Distillation of Language Models

arXiv 2603.07079 ↗First posted 7 Mar 2026cs.LG cs.CLcited by 151

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

On-policy distillation is a promising approach for transferring knowledge between language models, where a student learns from dense token-level signals along its own trajectories. This framework typically uses reverse KL divergence, encouraging the student to match the teacher's high-confidence predictions. However, we show that the mode-seeking property of reverse KL reduces generation diversity and yields unstable learning signals when the teacher distribution has high entropy. To address this, we introduce Entropy-Aware On-Policy Distillation. Our key idea is augmenting the standard reverse KL objective with forward KL when teacher entropy is high, capturing the full range of plausible outputs while retaining precise imitation elsewhere. It balances mode-seeking precision with mode-covering robustness without sacrificing on-policy training efficiency. Experiments show that our method maintains generation diversity (sustained token-level entropy) and improves student-teacher alignment (lower forward KL on high-entropy tokens). Across six math reasoning benchmarks, this yields Pass@8 accuracy gains of +1.37 for Qwen3-0.6B-Base, +2.39 for Qwen3-1.7B-Base, and +5.05 for Qwen3-4B-Base compared to baseline on-policy distillation methods. These results demonstrate that accounting for teacher uncertainty is essential for maintaining diversity and achieving effective knowledge transfer.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

42 of 54 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
30 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  13. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  14. 2016Sequence-Level Knowledge Distillationcited by 761
  15. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  16. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  17. 2015Policy Distillationcited by 414
  18. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  19. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  20. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  21. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  22. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  23. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  24. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  25. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  26. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 25
  27. 2025ToDi: Token-wise Distillation via Fine-Grained Divergence Controlcited by 23
  28. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  29. 2024FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillationcited by 11
  30. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 151.

151 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  3. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  4. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  5. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
  6. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 38
  7. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
  8. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  9. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  10. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  11. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  12. 2026Trust Region On-Policy Distillationcited by 26
38 more
  1. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  2. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  3. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
  4. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  5. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
  6. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  7. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
  8. 2026Trajectory-Refined Distillationcited by 14
  9. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 14
  10. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulationscited by 14
  11. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  12. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  13. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  14. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  15. 2026Rubric-based On-policy Distillationcited by 11
  16. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  17. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  18. 2026On-policy Distillation with Verifiable Rewardcited by 10
  19. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 9
  20. 2026OPRD: On-Policy Representation Distillationcited by 9
  21. 2026DOPD: Dual On-policy Distillationcited by 9
  22. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  23. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8
  24. 2026On-Policy Delta Distillationcited by 8
  25. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillationcited by 8
  26. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  27. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  28. 2026Trust-Region Behavior Blending for On-Policy Distillationcited by 7
  29. 2026PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformationcited by 7
  30. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rolloutscited by 7
  31. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Taskscited by 7
  32. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
  33. 2026Weak-to-Strong On-Policy Distillationcited by 7
  34. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  35. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  36. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 7
  37. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  38. 2026DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Modelscited by 6

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning What to Imitate: Entropy-Aware Distribution Mixing
  2. 2026E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
  3. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  4. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
  5. 2026Hybrid Policy Distillation for LLMs
  6. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
  8. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models