mlsanityData to 7 Oct 2026

Dynamic Dual-Granularity Skill Bank for Agentic RL

arXiv 2603.28716 ↗First posted 30 Mar 2026cs.AIcited by 33

On the map: LLM agents and tool use → Agent skill learning and evolution

Abstract (arXiv)

Agentic RL can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld, WebShop, and Search-Augmented QA tasks show that D2Skill substantially improves performance over skill-free baselines across models of different scales. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use. Builds on 45 of its 49 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

45 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning17 Mar
  2. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning9 Feb

Other references · 43

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  8. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  9. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  10. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  11. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  12. 2024Agent Workflow Memorycited by 342
31 more
  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  3. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  4. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  5. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  6. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  7. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 146
  8. 2026SimpleMem: Efficient Lifelong Memory for LLM Agentscited by 140
  9. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  10. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  11. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  12. 2026MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memorycited by 113
  13. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  14. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  15. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  16. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 77
  17. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  18. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  19. 2026Memento-Skills: Let Agents Design Agentscited by 56
  20. 2025Training-Free Group Relative Policy Optimizationcited by 53
  21. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  22. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
  23. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 39
  24. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  25. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 29
  26. 2025Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Taskscited by 22
  27. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 17
  28. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  29. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  30. 2026PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Reviewcited by 3
  31. 2026Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 33.

33 papers.

Sort:

By area:

  1. 2026SkillOS: Learning Skill Curation for Self-Evolving Agentscited by 45
  2. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  3. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  4. 2026Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agentscited by 7
  5. 2026CODESKILL: Learning Self-Evolving Skills for Coding Agentscited by 6
  6. 2026SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignmentcited by 6
  7. 2026Agent Skill Evaluation and Evolution: Frameworks and Benchmarkscited by 6
  8. 2026SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Trainingcited by 5
  9. 2026Co-Evolving Skill Generation and Policy Optimizationcited by 5
  10. 2026SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systemscited by 4
  11. 2026MIND-Skill: Quality-Guaranteed Skill Generation via Multi-Agent Induction and Deductioncited by 2
  12. 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolutioncited by 2
21 more
  1. 2026STRIDE: A Self-Reflective Agent Framework for Reliable Automatic Equation Discoverycited by 2
  2. 2026daVinci-kernel: Co-Evolving Skill Selection, Summarization, and Utilization via RL for GPU Kernel Optimizationcited by 2
  3. 2026Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learningcited by 1
  4. 2026Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systemscited by 1
  5. 2026MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agentscited by 1
  6. 2026HiSkill: Empowering LLM Agents with Hierarchical Skill Graphscited by 1
  7. 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentscited by 1
  8. 2026You Live More Than Once: Towards Hierarchical Skill Meta-Evolvingcited by 0
  9. 2026Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modelingcited by 0
  10. 2026SkillChain: Closing the Loop on Skill Evolution for Image-Based E-Commerce AI Assistantscited by 0
  11. 2026SkillMentor: LLM Agent Self-Evolution via Learning Blind-Spot Diagnosiscited by 0
  12. 2026Search2Skill: Skill Distillation Beyond Knowledge Boundaries Via Rubric-Based Reinforcement Learningcited by 0
  13. 2026Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimizationcited by 0
  14. 2026Coalition-Aware Skill Reliability for Self-Evolving Agentscited by 0
  15. 2026Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learnercited by 0
  16. 2026Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Securitycited by 0
  17. 2026CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolutioncited by 0
  18. 2026Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotationcited by 0
  19. 2026COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimizationcited by 0
  20. 2026UpliftMem: Learning Set-Level Uplift for Agent Memory Retrievalcited by 0
  21. 2026SkillFM: Generating Skills for LLM Agents via Latent Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SkillLens: Adaptive Multi-Granularity Skill Reuse for Cost-Efficient LLM Agents
  2. 2026SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles
  3. 2026ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL
  4. 2026Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
  5. 2026Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning
  6. 2026Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents
  7. 2026ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?
  8. 2026Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning