mlsanityData to 14 Oct 2026

Self-Distilled Agentic Reinforcement Learning

arXiv 2605.15155 ↗First posted 14 May 2026cs.LG cs.AI cs.CLcited by 55

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 48 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

42 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  2. 2025Agentic Reinforced Policy Optimization26 Jul
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 38

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
26 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  6. 2025Kimi K2: Open Agentic Intelligencecited by 567
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  8. 2019Learning by Cheatingcited by 383
  9. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
  10. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  11. 2026On-Policy Context Distillation for Language Modelscited by 186
  12. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  13. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  14. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  15. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  16. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  17. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  18. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47
  19. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  20. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  21. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  22. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  23. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  24. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  25. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  26. 2026UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 55.

55 papers.

Sort:

By area:

  1. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 32
  2. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 18
  3. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 12
  4. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 12
  5. 2026Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learningcited by 9
  6. 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 8
  7. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillationcited by 7
  8. 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agentscited by 6
  9. 2026TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learningcited by 6
  10. 2026Group-Reflective Self-Distillation for Agentic Reinforcement Learningcited by 6
  11. 2026SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignmentcited by 6
  12. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 5
38 more
  1. 2026From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Searchcited by 5
  2. 2026SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolutioncited by 5
  3. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 5
  4. 2026Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agentscited by 4
  5. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimizationcited by 4
  6. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 3
  7. 2026Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RLcited by 3
  8. 2026PaperGym: Rubric-Centered Evolution for Research-Plan Generationcited by 3
  9. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningcited by 3
  10. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  11. 2026Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detectioncited by 2
  12. 2026EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learningcited by 2
  13. 2026AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RLcited by 2
  14. 2026TTPO: Test-Time Policy Optimizationcited by 2
  15. 2026Negative Self-Distillation: Learning to Reason by Avoiding Flawscited by 2
  16. 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentscited by 2
  17. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  18. 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learningcited by 1
  19. 2026Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightcited by 1
  20. 2026ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillationcited by 0
  21. 2026VERPO: Verified Evidence Regularized Policy Optimizationcited by 0
  22. 2026Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddingscited by 0
  23. 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agentscited by 0
  24. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agentscited by 0
  25. 2026IGSD: Environment-Verified Hindsight Self-Distillation for Search Agentscited by 0
  26. 2026X-Tree: Tokenizing Reusable Experience for Efficient Agent Generalizationcited by 0
  27. 2026UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agentscited by 0
  28. 2026SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agentscited by 0
  29. 2026GenMem: Generative Symbolic Memory for Self-Evolving Harnesscited by 0
  30. 2026UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learningcited by 0
  31. 2026TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RLcited by 0
  32. 2026PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agentscited by 0
  33. 2026Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnessescited by 0
  34. 2026EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Makingcited by 0
  35. 2026ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agentscited by 0
  36. 2026PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agentscited by 0
  37. 2026SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learningcited by 0
  38. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Reinforcement Learning with Self-Distilled Reward Shaping
  2. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
  3. 2026Self-Distilled Policy Gradient
  4. 2026ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation
  5. 2026PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
  6. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation
  7. 2026RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
  8. 2026SR-OPSD: Self-Referenced On-Policy Self-Distillation