mlsanityData to 14 Oct 2026

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

arXiv 2604.10674 ↗First posted 12 Apr 2026cs.LG cs.AI cs.CLcited by 90

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 50 references that are in the map. First posted on arXiv 12 Apr 2026.

Builds on

43 of 50 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026OpenClaw-RL: Train Any Agent Simply by Talking10 Mar
  2. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning18 Nov
  3. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework5 Oct
  4. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning1 Oct
  5. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning10 Sep
  6. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  8. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2018Born Again Neural Networkscited by 546
  11. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  12. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
22 more
  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2025Learning to Reason under Off-Policy Guidancecited by 281
  3. 2026On-Policy Context Distillation for Language Modelscited by 186
  4. 2024Distilling System 2 into System 1cited by 159
  5. 2022Learning by Distilling Contextcited by 138
  6. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 137
  7. 2026Privileged Information Distillation for Language Modelscited by 90
  8. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 85
  9. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  10. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  11. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learningcited by 41
  12. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoningcited by 35
  13. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 34
  14. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  15. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  16. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 29
  17. 2026Online Experiential Learning for Language Modelscited by 28
  18. 2026MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environmentscited by 18
  19. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14
  20. 2025Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewritingcited by 13
  21. 2026UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agentscited by 12
  22. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 90.

90 papers.

Sort:

By area:

  1. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  2. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 32
  3. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 32
  4. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 32
  5. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 27
  6. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  7. 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 19
  8. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 18
  9. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learningcited by 16
  10. 2026Trajectory-Refined Distillationcited by 16
  11. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 12
  12. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 12
38 more
  1. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 12
  2. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 11
  3. 2026SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditingcited by 11
  4. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agentscited by 10
  5. 2026Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learningcited by 9
  6. 2026Rethinking Privileged Information in On-Policy Self-Distillationcited by 9
  7. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 8
  8. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 8
  9. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 7
  10. 2026Latent On-Policy Self-Distillationcited by 7
  11. 2026Contextual Agentic Memory is a Memo, Not True Memorycited by 6
  12. 2026SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestrationcited by 6
  13. 2026SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Trainingcited by 6
  14. 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agentscited by 6
  15. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 6
  16. 2026Procedural Memory Distillation: Online Reflection for Self-Improving Language Modelscited by 6
  17. 2026Group-Reflective Self-Distillation for Agentic Reinforcement Learningcited by 6
  18. 2026SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignmentcited by 6
  19. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillationcited by 5
  20. 2026Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memorycited by 4
  21. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  22. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimizationcited by 4
  23. 2026Parametric Skillscited by 3
  24. 2026Sample-Efficient Learning from Agent Experiencecited by 3
  25. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  26. 2026SKILL-KD: Contrastive Skill Distillation for LLM Agentscited by 3
  27. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningcited by 3
  28. 2026TimeClaw: A Time-Series AI Agent with Exploratory Execution Learningcited by 2
  29. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 2
  30. 2026From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agentscited by 2
  31. 2026Progressive Agent Skill Generation via Reinforcement Learningcited by 2
  32. 2026AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RLcited by 2
  33. 2026OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillationcited by 1
  34. 2026TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoningcited by 1
  35. 2026Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learningcited by 1
  36. 2026Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent Costcited by 1
  37. 2026Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agentscited by 1
  38. 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learningcited by 1

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
  2. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
  3. 2026Agentic Reinforcement Learning with Self-Distilled Reward Shaping
  4. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks
  5. 2026Building Multi-Task Agentic LLMs via Two-Phase Distillation
  6. 2026SAPD: Step-Aligned Privileged Distillation
  7. 2026PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
  8. 2026Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics