# Trajectory-Refined Distillation

arXiv: https://arxiv.org/abs/2606.08432 · HTML: https://mlsanity.com/paper/2606.08432/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 34 references that are in the map.
- First posted on arXiv 7 Jun 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 32)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning](https://mlsanity.com/paper/2504.13914/) (2025)
- [TACO: Topics in Algorithmic COde generation dataset](https://mlsanity.com/paper/2312.14852/) (2023)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [TIP: Token Importance in On-Policy Distillation](https://mlsanity.com/paper/2604.14084/) (2026)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [Stable On-Policy Distillation through Adaptive Target Reformulation](https://mlsanity.com/paper/2601.07155/) (2026)
- [AMO-Bench: Large Language Models Still Struggle in High School Math Competitions](https://mlsanity.com/paper/2510.26768/) (2025)
- [GATES: Self-Distillation under Privileged Context with Consensus Gating](https://mlsanity.com/paper/2602.20574/) (2026)
- [Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models](https://mlsanity.com/paper/2604.08527/) (2026)

## Built on by (most cited first, 16 of 16)

- [Pass the Baton: Trajectory-Relayed On-Policy Distillation](https://mlsanity.com/paper/2607.26057/) (2026)
- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning](https://mlsanity.com/paper/2608.16333/) (2026)
- [RISE: Recursive Improvement via Self-Extrapolating Policy Distillation](https://mlsanity.com/paper/2609.05295/) (2026)
- [Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress](https://mlsanity.com/paper/2608.19408/) (2026)
- [Negative Self-Distillation: Learning to Reason by Avoiding Flaws](https://mlsanity.com/paper/2609.11699/) (2026)
- [Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation](https://mlsanity.com/paper/2609.22254/) (2026)
- [TISD: On-Policy Self-Distillation with Trajectory Intervention](https://mlsanity.com/paper/2609.30878/) (2026)
- [OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation](https://mlsanity.com/paper/2609.16459/) (2026)
- [UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents](https://mlsanity.com/paper/2609.34036/) (2026)
- [SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models](https://mlsanity.com/paper/2609.34044/) (2026)
- [Learning from Teacher Continuations at Student States](https://mlsanity.com/paper/2609.36246/) (2026)
- [From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation](https://mlsanity.com/paper/2609.37510/) (2026)
- [How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation](https://mlsanity.com/paper/2610.04950/) (2026)
- [When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better](https://mlsanity.com/paper/2610.11291/) (2026)
- [DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision](https://mlsanity.com/paper/2610.19302/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts. In this work, we identify a common structural cause underlying OPD, which we call prefix failure. Under prefix failure, dense per-token supervision induces a bimodal teacher mixture and fragmented gradients that token-level loss truncation or reweighting fail to address. We thus propose Trajectory-Refined Distillation (TRD), a trajectory-level correction method that revises the student's rollout under the teacher guidance while within on-policy support. By correcting problematic prefixes before distillation, TRD mitigates prefix failure at its source. Moreover, TRD improves exploration by exposing the student to alternative valid derivations under teacher guidance, even when the original rollouts are already correct. TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged information as the teacher. TRD improves average math accuracy while remaining competitive on code. Under OPSD, it solves 9/23 AMOBench questions left unsolved by the Qwen3-8B base model, nearly doubling the strongest dense-KL baseline. Code is available at https://github.com/louieworth/trd.