# When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

arXiv: https://arxiv.org/abs/2608.05219 · HTML: https://mlsanity.com/paper/2608.05219/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents](https://mlsanity.com/paper/2604.24005/) (2026)
- [ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflection](https://mlsanity.com/paper/2505.15182/) (2025)
- [StateAct: Enhancing LLM Base Agents via Self-prompting and State-tracking](https://mlsanity.com/paper/2410.02810/) (2024)
- [TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training](https://mlsanity.com/paper/2607.05804/) (2026)
- [SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation](https://mlsanity.com/paper/2606.19659/) (2026)
- [Multi-Turn On-Policy Distillation with Prefix Replay](https://mlsanity.com/paper/2607.04763/) (2026)
- [What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents](https://mlsanity.com/paper/2605.19447/) (2026)
- [Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance](https://mlsanity.com/paper/2606.00305/) (2026)
- [HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents](https://mlsanity.com/paper/2605.17873/) (2026)
- [From History to State: Constant-Context Skill Learning for LLM Agents](https://mlsanity.com/paper/2605.05413/) (2026)

## Built on by (most cited first, 5 of 5)

- [RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.20784/) (2026)
- [Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention](https://mlsanity.com/paper/2609.39494/) (2026)
- [ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience](https://mlsanity.com/paper/2610.05303/) (2026)
- [GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents](https://mlsanity.com/paper/2610.08959/) (2026)
- [State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents](https://mlsanity.com/paper/2610.13238/) (2026)

## Abstract (arXiv)

Multi-turn agents trained with sparse outcome rewards receive limited supervision for intermediate decisions. Privileged on-policy self-distillation provides denser guidance by having a synchronized teacher re-score student responses with training-only references. However, the student may take different actions or complete subgoals in another order, reaching states for which the reference provides no applicable continuation. The resulting state--reference mismatch can lead the reference-conditioned teacher to penalize valid actions. We introduce State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which uses the student's reached state to decide when and how a reference should guide the student. At each turn, SMRC-SD checks state compatibility with the reference, distills only on matched turns, and contextualizes the reference by augmenting it with a reached-state summary and a compatible continuation. At inference, the policy requires neither references nor state matching. Experiments spanning three model sizes show gains over unconditional full-path self-distillation in embodied interaction (ALFWorld), web navigation (WebShop), and tool use (FTRL). FTRL-trained policies also transfer to BFCL and ToolHop without additional training. Code is available at https://github.com/liujunzhuo/SMRC-SD.