mlsanityData to 14 Oct 2026

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

arXiv 2610.03185 ↗First posted 2 Oct 2026cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification also fits: Reward modeling and RLHF

also in 2 more
Abstract (arXiv)

On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 31 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

27 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking2 Feb

Other references · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  6. 2016Sequence-Level Knowledge Distillationcited by 801
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  10. 2022Large Language Models Are Reasoning Teacherscited by 329
  11. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  12. 2026MiMo-V2-Flash Technical Reportcited by 191
14 more
  1. 2026On-Policy Context Distillation for Language Modelscited by 186
  2. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  3. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  4. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
  5. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  6. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  7. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
  8. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
  9. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 30
  10. 2023Understanding In-Context Learning from Repetitionscited by 29
  11. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22
  12. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 8
  13. 2026SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoningcited by 6
  14. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
  2. 2026Reward-Aligned Reweighting for On-Policy Distillation
  3. 2026When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
  4. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
  5. 2026On the Off-Policy Teacher in On-Policy Distillation
  6. 2026Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?
  7. 2026One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation
  8. 2026Correcting or Continuing? What On-Policy Distillation Learns from Student Errors