# Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts

arXiv: https://arxiv.org/abs/2606.21994 · HTML: https://mlsanity.com/paper/2606.21994/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 27 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [TIP: Token Importance in On-Policy Distillation](https://mlsanity.com/paper/2604.14084/) (2026)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [JustRL: Scaling a 1.5B LLM with a Simple RL Recipe](https://mlsanity.com/paper/2512.16649/) (2025)
- [Stable On-Policy Distillation through Adaptive Target Reformulation](https://mlsanity.com/paper/2601.07155/) (2026)
- [Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation](https://mlsanity.com/paper/2605.26844/) (2026)
- [SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting](https://mlsanity.com/paper/2604.10688/) (2026)
- [Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation](https://mlsanity.com/paper/2604.13010/) (2026)
- [When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2605.21606/) (2026)
- [Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning](https://mlsanity.com/paper/2605.07804/) (2026)
- [Are Full Rollouts Necessary for On-Policy Distillation?](https://mlsanity.com/paper/2605.31490/) (2026)

## Built on by (most cited first, 10 of 10)

- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation](https://mlsanity.com/paper/2608.03092/) (2026)
- [ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation](https://mlsanity.com/paper/2608.10905/) (2026)
- [1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation](https://mlsanity.com/paper/2609.24432/) (2026)
- [Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation](https://mlsanity.com/paper/2607.29494/) (2026)
- [Distillation as Probability Transport: Routed On-Policy Distillation](https://mlsanity.com/paper/2609.08337/) (2026)
- [Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation](https://mlsanity.com/paper/2610.02678/) (2026)
- [Flash-OPD: Fast On-Policy Distillation](https://mlsanity.com/paper/2610.06105/) (2026)
- [DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation](https://mlsanity.com/paper/2610.11659/) (2026)
- [PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining](https://mlsanity.com/paper/2610.19601/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories. However, scaling OPD to long-horizon reasoning exposes a reliability and efficiency problem: standard OPD assigns every candidate the same long rollout budget, even though some trajectories may quickly become weakly aligned with the teacher and provide less useful supervision. Prior analyses suggest that teacher--student compatibility is important for OPD success, motivating early-prefix top-k overlap as a proxy for continuation value. Continuing low-overlap trajectories to the full rollout length may cause them to drift further from the teacher as generation proceeds, increasing computational cost while providing limited distillation benefit. To address this, we introduce Prefix-Guided On-Policy Distillation (PG-OPD), a rollout-allocation framework that estimates continuation value from fixed-length prefixes. PG-OPD computes teacher-student top-k overlap in an early probe window and allocates long rollouts only to high-overlap candidates, while stopping the rest at the prefix length. Across teacher--student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD achieves up to a 4.80-point accuracy gain and up to a 2.46x wall-clock speedup across configurations. These results demonstrate that using early-prefix compatibility to guide candidate pruning can improve both training efficiency and reasoning performance.