# ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation arXiv: https://arxiv.org/abs/2607.13124 · HTML: https://mlsanity.com/paper/2607.13124/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. - Builds on 46 of its 51 references that are in the map. - First posted on arXiv 14 Jul 2026. ## Placed in - Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/) ## Builds on (most cited first, 30 of 46) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021) - [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025) - [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023) - [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024) - [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023) - [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023) - [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023) - [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015) - [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023) - [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024) - [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024) - [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024) - [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024) - [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024) - [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023) - [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024) ## Built on by (most cited first, 8 of 8) - [Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation](https://mlsanity.com/paper/2608.01953/) (2026) - [Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.23697/) (2026) - [On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment](https://mlsanity.com/paper/2607.27081/) (2026) - [Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation](https://mlsanity.com/paper/2607.29494/) (2026) - [IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models](https://mlsanity.com/paper/2607.15732/) (2026) - [SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models](https://mlsanity.com/paper/2609.34044/) (2026) - [TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery](https://mlsanity.com/paper/2610.09074/) (2026) - [ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery](https://mlsanity.com/paper/2610.11332/) (2026) ## Abstract (arXiv) Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires. Two observations trace this gap. First, greedy textscpass@1 nearly vanishes after compression, yet textscpass@k recovers substantially under repeated sampling: useful generations are demoted, not erased. Second, the recoverable regime fails mainly through suffix repetition. Recovery should therefore train on the compressed model's own on-policy states with dense token-level supervision, which On-Policy Distillation (OPD) provides by reusing the pre-compression model as a frozen teacher. However, long on-policy rollouts spend early recovery budget on low-information repetitive suffixes, delaying loss descent. To mitigate this waste, we propose shortopd, a short-to-long OPD schedule that detects teacher-confirmed repetitive suffixes, treats the surviving prefix as each rollout's effective length, and allocates future rollout budgets to the effective lengths the policy can currently use. Across math, code, and open-ended generation, shortopd raises the compressed model's score to about 9× its unrecovered value and 1.6--4.4× standard recovery recipes (SFT w/o KD, KD, and SeqKD), and it matches a fixed 8192-token rollout horizon within two points using a quarter of the training time (8.5 vs. 35.9 hours) and 71% fewer rollout tokens. We hope this recipe helps move structured pruning beyond marginal gains on perplexity and multiple-choice benchmarks, a step closer to deployment-ready generation quality.