# Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

arXiv: https://arxiv.org/abs/2605.07804 · HTML: https://mlsanity.com/paper/2605.07804/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 28 of 28)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Small Models Struggle to Learn from Strong Reasoners](https://mlsanity.com/paper/2502.12143/) (2025)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling](https://mlsanity.com/paper/2407.09887/) (2024)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)
- [Stable On-Policy Distillation through Adaptive Target Reformulation](https://mlsanity.com/paper/2601.07155/) (2026)
- [Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration](https://mlsanity.com/paper/2508.13755/) (2025)
- [HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation](https://mlsanity.com/paper/2603.23871/) (2026)
- [How Far Can Unsupervised RLVR Scale LLM Training?](https://mlsanity.com/paper/2603.08660/) (2026)
- [Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning](https://mlsanity.com/paper/2602.03249/) (2026)
- [EHRAG: Bridging Semantic Gaps in Lightweight GraphRAG via Hybrid Hypergraph Construction and Retrieval](https://mlsanity.com/paper/2604.17458/) (2026)

## Built on by (most cited first, 27 of 27)

- [Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation](https://mlsanity.com/paper/2608.01953/) (2026)
- [Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts](https://mlsanity.com/paper/2606.21994/) (2026)
- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection](https://mlsanity.com/paper/2609.05198/) (2026)
- [When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation](https://mlsanity.com/paper/2609.20511/) (2026)
- [Simple-OPD: Demystifying Warm-up for On-policy Distillation](https://mlsanity.com/paper/2608.06802/) (2026)
- [Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.08176/) (2026)
- [ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation](https://mlsanity.com/paper/2608.10905/) (2026)
- [REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation](https://mlsanity.com/paper/2608.11698/) (2026)
- [When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation](https://mlsanity.com/paper/2608.27960/) (2026)
- [1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation](https://mlsanity.com/paper/2609.24432/) (2026)
- [On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment](https://mlsanity.com/paper/2607.27081/) (2026)
- [Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation](https://mlsanity.com/paper/2607.29494/) (2026)
- [Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation](https://mlsanity.com/paper/2609.14636/) (2026)
- [Blockwise Policy-Drift Gating for On-Policy Distillation](https://mlsanity.com/paper/2606.24084/) (2026)
- [Binocular Gaze Estimation with Single Camera and Single Light Source](https://mlsanity.com/paper/2607.05473/) (2026)
- [Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning](https://mlsanity.com/paper/2607.27888/) (2026)
- [AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discovery](https://mlsanity.com/paper/2608.01303/) (2026)
- [RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents](https://mlsanity.com/paper/2609.15684/) (2026)
- [UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents](https://mlsanity.com/paper/2609.34036/) (2026)
- [On the Off-Policy Teacher in On-Policy Distillation](https://mlsanity.com/paper/2609.38360/) (2026)
- [Diagnosing On-Policy Self-Distillation for Reasoning Language Models](https://mlsanity.com/paper/2609.39118/) (2026)
- [Outcome-Guided On-Policy Self-Distillation](https://mlsanity.com/paper/2610.05070/) (2026)
- [Flash-OPD: Fast On-Policy Distillation](https://mlsanity.com/paper/2610.06105/) (2026)
- [DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision](https://mlsanity.com/paper/2610.19302/) (2026)
- [Recovering Reliable Supervision via Critique-Guided On-Policy Distillation](https://mlsanity.com/paper/2610.20572/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) leverages dense teacher rewards to enhance reasoning models. However, scaling OPD to long-horizon tasks exposes a critical flaw: as the student's generated prefix inevitably diverges from the teacher's thought process, the teacher's dense reward loses local exploitability. Continuing to generate and evaluate tokens on these ``drifted'' trajectories not only degrades reward quality but also incurs massive computational waste. To address this, we introduce Prune-OPD, a framework that dynamically aligns training budgets with supervision quality. By continuously monitoring the local compatibility between student and teacher predictions (e.g., via top-k overlap), Prune-OPD detects prefix-drift events in real time. Upon detecting severe drift, it monotonically down-weights subsequent unreliable rewards and triggers dynamic rollout truncation. This allows the training process to halt futile generation and reallocate compute strictly to reliable teacher supervision. Across diverse teacher-student combinations, Prune-OPD consistently aligns computation with supervision reliability. When prefix drift makes dense teacher rewards unreliable, it reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks (AMC, AIME, HMMT). When student-teacher compatibility remains high, it automatically preserves long-context supervision by expanding the training window. These results suggest that Prune-OPD improves OPD not by blindly shortening rollouts, but by reallocating computation toward locally exploitable teacher rewards.