mlsanityData to 14 Oct 2026

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

arXiv 2606.15912 ↗First posted 14 Jun 2026cs.LG cs.AIcited by 27

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice. On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students, but we find that it suffers a characteristic failure mode in this setting: small student errors compound across turns and push the trajectory out of the teacher's familiar state distribution, so the teacher's supervision becomes least reliable precisely where the student needs it most. We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout and schedules the teacher's intervention probability along a curriculum that decays to zero. Strong guidance keeps early trajectories close to the teacher distribution and is then gradually withdrawn to recover the purely on-policy regime used at inference. On ALFWorld, ScienceWorld, and WebShop, distilling Qwen3 students from a Qwen3-30B-A3B teacher, Guided-OPD yields average relative gains of 21.1% in Score and 25.5% in Success Rate over vanilla OPD, with larger gains on smaller students.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 35 of its 48 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

35 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation8 May
  2. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe5 May
  3. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents27 Apr
  4. 2026TIP: Token Importance in On-Policy Distillation15 Apr
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  6. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  7. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing2 Apr
  8. 2026A Survey of On-Policy Distillation for Large Language Models1 Apr
  9. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes26 Mar
  10. 2026Entropy-Aware On-Policy Distillation of Language Models7 Mar
  11. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation12 Feb
  12. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes17 Jun

Other references · 23

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  11. 2023The False Promise of Imitating Proprietary LLMscited by 263
  12. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
11 more
  1. 2026MiMo-V2-Flash Technical Reportcited by 191
  2. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  3. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  4. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  5. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  6. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  7. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 47
  8. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  9. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  10. 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Modelscited by 9
  11. 2025AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switchingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 27.

27 papers.

Sort:

By area:

  1. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillationcited by 11
  2. 2026Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuningcited by 8
  3. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimizationcited by 4
  4. 2026EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Modelscited by 3
  5. 2026DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillationcited by 3
  6. 2026ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillationcited by 3
  7. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningcited by 3
  8. 2026From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agentscited by 2
  9. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  10. 2026Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillationcited by 0
  11. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agentscited by 0
  12. 2026Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillationcited by 0
15 more
  1. 2026UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agentscited by 0
  2. 2026DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agentscited by 0
  3. 2026Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agentscited by 0
  4. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0
  5. 2026Inductive Feedback for Mixed-Policy Distillationcited by 0
  6. 2026An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoningcited by 0
  7. 2026Learning from Teacher Continuations at Student Statescited by 0
  8. 2026Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamicscited by 0
  9. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RLcited by 0
  10. 2026Harness-Aware Distillation for Small Language Model Agentscited by 0
  11. 2026TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recoverycited by 0
  12. 2026State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agentscited by 0
  13. 2026DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervisioncited by 0
  14. 2026Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agentscited by 0
  15. 2026ActionOPD: Distilling Decisions, Not Tokenscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Turn On-Policy Distillation with Prefix Replay
  2. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
  3. 2026Stochastic Teacher Intervention for Agentic On-Policy Distillation
  4. 2026Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
  5. 2026Fast and Effective On-policy Distillation from Reasoning Prefixes
  6. 2026Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers
  7. 2026Interactive-Policy Distillation with Bidirectional Propose-and-Verify
  8. 2026Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?