# Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

arXiv: https://arxiv.org/abs/2606.00305 · HTML: https://mlsanity.com/paper/2606.00305/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 42 of its 90 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 42)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents](https://mlsanity.com/paper/2601.03236/) (2026)
- [Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe](https://mlsanity.com/paper/2605.03677/) (2026)
- [FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval](https://mlsanity.com/paper/2503.21309/) (2025)
- [DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models](https://mlsanity.com/paper/2505.13975/) (2025)
- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMA](https://mlsanity.com/paper/2512.12812/) (2025)
- [Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval](https://mlsanity.com/paper/2601.12768/) (2026)
- [SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models](https://mlsanity.com/paper/2601.03555/) (2026)
- [Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations](https://mlsanity.com/paper/2602.19320/) (2026)
- [Enhancing Financial Report Question-Answering: A Retrieval-Augmented Generation System with Reranking Analysis](https://mlsanity.com/paper/2603.16877/) (2026)
- [Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.19386/) (2026)
- [AutoNeural: Co-Designing Vision-Language Models for NPU Inference](https://mlsanity.com/paper/2512.02924/) (2025)
- [ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.20358/) (2026)
- [Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation](https://mlsanity.com/paper/2510.10925/) (2025)
- [Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models](https://mlsanity.com/paper/2603.21389/) (2026)
- [TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval](https://mlsanity.com/paper/2604.21806/) (2026)
- [Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict](https://mlsanity.com/paper/2605.14473/) (2026)
- [AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models](https://mlsanity.com/paper/2509.00039/) (2025)
- [HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2601.16155/) (2026)
- [Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG](https://mlsanity.com/paper/2605.28044/) (2026)
- [Sustainable Hybrid Document-Routed Retrieval for Financial RAG: Resolving the Robustness-Precision Trade-off](https://mlsanity.com/paper/2603.26815/) (2026)
- [MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Models](https://mlsanity.com/paper/2511.17448/) (2025)
- [Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation](https://mlsanity.com/paper/2605.09253/) (2026)
- [Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data](https://mlsanity.com/paper/2503.05587/) (2025)

## Built on by (most cited first, 13 of 13)

- [Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation](https://mlsanity.com/paper/2608.01953/) (2026)
- [SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation](https://mlsanity.com/paper/2608.04419/) (2026)
- [When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents](https://mlsanity.com/paper/2608.05219/) (2026)
- [BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering](https://mlsanity.com/paper/2609.06922/) (2026)
- [TISD: On-Policy Self-Distillation with Trajectory Intervention](https://mlsanity.com/paper/2609.30878/) (2026)
- [New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models](https://mlsanity.com/paper/2609.11022/) (2026)
- [KuaiRP Series Role-playing Models Technical Report](https://mlsanity.com/paper/2609.11127/) (2026)
- [ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training](https://mlsanity.com/paper/2609.16639/) (2026)
- [Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teachers](https://mlsanity.com/paper/2609.32674/) (2026)
- [Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents](https://mlsanity.com/paper/2609.33391/) (2026)
- [What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation](https://mlsanity.com/paper/2609.33455/) (2026)
- [Learning from Think-Mode Advantage via On-Policy Distillation](https://mlsanity.com/paper/2609.37044/) (2026)
- [DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation](https://mlsanity.com/paper/2610.11659/) (2026)

## Abstract (arXiv)

On-Policy Distillation (OPD) improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision. Although OPD operates on trajectories, its learning signal remains token-level: it identifies deviations through high-loss tokens and repairs them through local reverse-KL correction. We show that this "trajectory-sampled but token-learned" mechanism cannot reliably bridge student trajectories toward teacher trajectories. About 30% of high-loss tokens fall into the low-divergence regime, indicating that many are surface-form mismatches rather than real reasoning forks. Moreover, even truly divergent tokens are difficult to repair with isolated token-level supervision, since reasoning failures often unfold as short-horizon distributional drift. We propose Trajectory-aware OPD (TOPD), which uses near-future trajectory information to identify real divergent states and distribute guidance across multiple future tokens. Experiments show that suppressing non-divergent high-loss tokens improves standard OPD from 47.8% to 48.2% average accuracy, while TOPD further improves performance to 52.2%, with gains on AIME24 from 60.0% to 63.3% and AIME25 from 46.7% to 53.3%.