mlsanityData to 7 Oct 2026

Fast and Effective On-policy Distillation from Reasoning Prefixes

arXiv 2602.15260 ↗First posted 16 Feb 2026cs.LG cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for long responses. Our initial analysis shows that, during OPD, training signals are often concentrated in the prefix of each output, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

23 of 41 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2016Sequence-Level Knowledge Distillationcited by 761
11 more
  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  4. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  5. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  6. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  7. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  8. 2025The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Modelscited by 31
  9. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  10. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  11. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 43.

43 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  2. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  3. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  4. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  5. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  6. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  7. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  8. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  9. 2026Rubric-based On-policy Distillationcited by 11
  10. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  11. 2026Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillationcited by 9
  12. 2026DOPD: Dual On-policy Distillationcited by 9
31 more
  1. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  2. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8
  3. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillationcited by 8
  4. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  5. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  6. 2026Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillationcited by 5
  7. 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Controlcited by 4
  8. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Modelscited by 4
  9. 2026ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillationcited by 3
  10. 2026Simple-OPD: Demystifying Warm-up for On-policy Distillationcited by 3
  11. 2026When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillationcited by 3
  12. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  13. 2026From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuationcited by 1
  14. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  15. 20261% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillationcited by 1
  16. 2026AdaStop: Cost-Aware Early Stopping for DNN Test Selectioncited by 0
  17. 2026Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Updatecited by 0
  18. 2026ABOPD: Antibody CDR Design via On-Policy Distillationcited by 0
  19. 2026RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfercited by 0
  20. 2026Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillationcited by 0
  21. 2026Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillationcited by 0
  22. 2026Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillationcited by 0
  23. 2026What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillationcited by 0
  24. 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?cited by 0
  25. 2026Counterfactual Attention Policy Distillation for Temporal Video Groundingcited by 0
  26. 2026From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillationcited by 0
  27. 2026On the Off-Policy Teacher in On-Policy Distillationcited by 0
  28. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Modelscited by 0
  29. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retentioncited by 0
  30. 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoningcited by 0
  31. 2026DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillationcited by 0