mlsanityData to 14 Oct 2026

SOD: Step-wise On-policy Distillation for Small Language Model Agents

arXiv 2605.07725 ↗First posted 8 May 2026cs.CL cs.AIcited by 21

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Tool-integrated reasoning (TIR) is difficult to scale to small language models due to instability in long-horizon tool interactions and limited model capacity. While reinforcement learning methods like group relative policy optimization provide only sparse outcome-level rewards. Recently, on-policy distillation (OPD) has gained popularity by supplying dense token-level supervision from a teacher on student-generated trajectories. However, our experiments indicate that applying OPD to TIR leads to a critical failure mode: erroneous tool calls tend to cascade across subsequent reasoning steps, progressively amplifying student-teacher divergence and rendering the teacher's token-level supervision increasingly unreliable. To address this, we propose SOD, a step-wise on-policy distillation framework for small language model agents, which adaptively reweights distillation strength at each step based on step-level divergence. Therefore, SOD can attenuate potentially misleading teacher signals in high-divergence regions while preserving dense guidance in well-aligned states. Experiments on challenging math, science, and code benchmarks show that SOD achieves up to 20.86% improvement over the second-best baseline. Notably, our 0.6B student achieves 26.13% on AIME 2025, demonstrating effective transfer of agentic reasoning to lightweight models. Our code is available at https://github.com/YoungZ365/SOD.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 71 of its 75 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

71 of 75 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  2. 2026OpenClaw-RL: Train Any Agent Simply by Talking10 Mar
  3. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System2 Feb
  4. 2025Demystifying Reinforcement Learning in Agentic Reasoning13 Oct
  5. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  6. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning28 Apr
  7. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  8. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  9. 2025ToRL: Scaling Tool-Integrated RL30 Mar
  10. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb
  11. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 60

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
38 more
  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  6. 2015Policy Distillationcited by 440
  7. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  8. 2026Reinforcement Learning via Self-Distillationcited by 332
  9. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  10. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  11. 2023The False Promise of Imitating Proprietary LLMscited by 263
  12. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  13. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  14. 2026Self-Distillation Enables Continual Learningcited by 247
  15. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  16. 2026On-Policy Context Distillation for Language Modelscited by 186
  17. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  18. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  19. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  20. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  21. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  22. 2026Self-Distilled RLVRcited by 131
  23. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  24. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  25. 2026Privileged Information Distillation for Language Modelscited by 90
  26. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  27. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  28. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  29. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  30. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  31. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59
  32. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  33. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  34. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 53
  35. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 47
  36. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47
  37. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  38. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 43

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 21.

21 papers.

Sort:

By area:

  1. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 17
  2. 2026Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuningcited by 8
  3. 2026SAF-OPD: Stable Advantage Fusion for On-Policy Distillationcited by 5
  4. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  5. 2026Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillationcited by 4
  6. 2026DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillationcited by 3
  7. 2026TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoningcited by 1
  8. 2026From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasonerscited by 0
  9. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agentscited by 0
  10. 2026UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agentscited by 0
  11. 2026USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agentscited by 0
  12. 2026MAS-OPD: On-Policy Distillation for Multi-agent Systemscited by 0
9 more
  1. 2026Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agentscited by 0
  2. 2026Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnessescited by 0
  3. 2026Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RLcited by 0
  4. 2026On the Off-Policy Teacher in On-Policy Distillationcited by 0
  5. 2026PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agentscited by 0
  6. 2026GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agentscited by 0
  7. 2026SAPD: Step-Aligned Privileged Distillationcited by 0
  8. 2026LLM Value Functions for Credit Assignment in Agentic RLcited by 0
  9. 2026DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  3. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  4. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
  5. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  6. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
  7. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  8. 2026EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models