mlsanityData to 7 Oct 2026

Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

arXiv 2602.12125 ↗First posted 12 Feb 2026cs.LG cs.AI cs.CLcited by 157

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

On-policy distillation (OPD), which aligns the student with the teacher's logit distribution on student-generated trajectories, has demonstrated strong empirical gains in improving student performance and often outperforms off-policy distillation and reinforcement learning (RL) paradigms. In this work, we first theoretically show that OPD is a special case of dense KL-constrained RL where the reward function and the KL regularization are always weighted equally and the reference model can by any model. Then, we propose the Generalized On-Policy Distillation (G-OPD) framework, which extends the standard OPD objective by introducing a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization. Through comprehensive experiments on math reasoning and code generation tasks, we derive two novel insights: (1) Setting the reward scaling factor to be greater than 1 (i.e., reward extrapolation), which we term ExOPD, consistently improves over standard OPD across a range of teacher-student size pairings. In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, back into the original student, ExOPD enables the student to even surpass the teacher's performance boundary and outperform the domain teachers. (2) Building on ExOPD, we further find that in the strong-to-weak distillation setting (i.e., distilling a smaller student from a larger teacher), performing reward correction by choosing the reference model as the teacher's base model before RL yields a more accurate reward signal and further improves distillation performance. However, this choice assumes access to the teacher's pre-RL variant and incurs more computational overhead. We hope our work offers new insights for future research on OPD.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). Builds on 37 of its 44 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

37 of 44 references.

References · 37

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2025LIMO: Less is More for Reasoningcited by 488
25 more
  1. 2025Process Reinforcement through Implicit Rewardscited by 478
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  3. 2015Policy Distillationcited by 414
  4. 2024InternLM2 Technical Reportcited by 391
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  6. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  7. 2026Reinforcement Learning via Self-Distillationcited by 285
  8. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  10. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  11. 2026Self-Distillation Enables Continual Learningcited by 212
  12. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  13. 2026MiMo-V2-Flash Technical Reportcited by 163
  14. 2024Free Process Rewards without Process Labelscited by 162
  15. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  16. 2026Privileged Information Distillation for Language Modelscited by 79
  17. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  18. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  19. 2024Model Extrapolation Expedites Alignmentcited by 37
  20. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  21. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  22. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  23. 2023Distilling Rule-based Knowledge into Large Language Modelscited by 9
  24. 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewardingcited by 9
  25. 2026ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 157.

157 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  3. 2026Self-Distilled Agentic Reinforcement Learningcited by 49
  4. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  5. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
  6. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 39
  7. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  8. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  9. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  10. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  11. 2026Trust Region On-Policy Distillationcited by 26
  12. 2026On the Position Bias of On-Policy Distillationcited by 23
38 more
  1. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
  2. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 21
  3. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  4. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  5. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  6. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 14
  7. 2026Trajectory-Refined Distillationcited by 14
  8. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 14
  9. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulationscited by 14
  10. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  11. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  12. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Samplingcited by 12
  13. 2026DanceOPD: On-Policy Generative Field Distillationcited by 12
  14. 2026Rubric-based On-policy Distillationcited by 11
  15. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  16. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  17. 2026Crosslingual On-Policy Self-Distillation for Multilingual Reasoningcited by 10
  18. 2026On-policy Distillation with Verifiable Rewardcited by 10
  19. 2026From Generic Correlation to Input-Specific Credit in On-Policy Self Distillationcited by 9
  20. 2026OPRD: On-Policy Representation Distillationcited by 9
  21. 2026DOPD: Dual On-policy Distillationcited by 9
  22. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  23. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  24. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8
  25. 2026On-Policy Delta Distillationcited by 8
  26. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 8
  27. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  28. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Taskscited by 7
  29. 2026H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillationcited by 7
  30. 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 7
  31. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
  32. 2026Weak-to-Strong On-Policy Distillationcited by 7
  33. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  34. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  35. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 5
  36. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  37. 2026Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillationcited by 5
  38. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-Gated On-Policy Distillation
  2. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillation
  3. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
  4. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  5. 2026Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
  6. 2026DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation
  7. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  8. 2026Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers