mlsanityData to 14 Oct 2026

Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models

arXiv 2604.08527 ↗First posted 9 Apr 2026cs.CL cs.LGcited by 30

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 28 of its 30 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

28 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  2. 2026Reinforcement Learning via Self-Distillation28 Jan
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
13 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2016Sequence-Level Knowledge Distillationcited by 801
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  5. 2025Process Reinforcement through Implicit Rewardscited by 504
  6. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  7. 2015Policy Distillationcited by 440
  8. 2025Learning to Reason under Off-Policy Guidancecited by 281
  9. 2026MiMo-V2-Flash Technical Reportcited by 191
  10. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  11. 2019Distilling Policy Distillationcited by 84
  12. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  13. 2020Dual Policy Distillationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 30.

30 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  2. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
  3. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 32
  4. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 20
  5. 2026Trajectory-Refined Distillationcited by 16
  6. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 15
  7. 2026DanceOPD: On-Policy Generative Field Distillationcited by 14
  8. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  9. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 10
  10. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 7
  11. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 7
  12. 2026SAF-OPD: Stable Advantage Fusion for On-Policy Distillationcited by 5
18 more
  1. 2026When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillationcited by 5
  2. 2026Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillationcited by 4
  3. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
  4. 2026SEER: Long-Context Reasoning via Selective Visual-Text Compressioncited by 3
  5. 2026Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learningcited by 2
  6. 2026What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillationcited by 2
  7. 2026Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoderscited by 1
  8. 2026Blockwise Policy-Drift Gating for On-Policy Distillationcited by 0
  9. 2026SR-OPSD: Self-Referenced On-Policy Self-Distillationcited by 0
  10. 2026Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Taskscited by 0
  11. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectivescited by 0
  12. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Modelscited by 0
  13. 2026Learning to Revise Reasoning with Segment-wise On-Policy Distillationcited by 0
  14. 2026Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspectivecited by 0
  15. 2026When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Bettercited by 0
  16. 2026Branching On-Policy Distillationcited by 0
  17. 2026Token Utility for On-Policy Distillation via Local Trust-Region Optimizationcited by 0
  18. 2026DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
  2. 2026Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
  3. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Example
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
  5. 2026Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
  6. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  7. 2026GFD-OPD: Guidance-Folded On-Policy Distillation of Diffusion Models Across Scales
  8. 2026KL for a KL: On-Policy Distillation with Control Variate Baseline