# SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

arXiv: https://arxiv.org/abs/2602.08234 · HTML: https://mlsanity.com/paper/2602.08234/

## Key facts

- Cited by 278 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 30 of 39)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [SimpleMem: Efficient Lifelong Memory for LLM Agents](https://mlsanity.com/paper/2601.02553/) (2026)
- [EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle](https://mlsanity.com/paper/2510.16079/) (2025)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)

## Built on by (most cited first, 30 of 278)

- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://mlsanity.com/paper/2604.25850/) (2026)
- [CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification](https://mlsanity.com/paper/2604.01687/) (2026)
- [SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization](https://mlsanity.com/paper/2604.02268/) (2026)
- [SkillClaw: Let Skills Evolve Collectively with Agentic Evolver](https://mlsanity.com/paper/2604.08377/) (2026)
- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026)
- [Self-Distilled Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.15155/) (2026)
- [SkillNet: Create, Evaluate, and Connect AI Skills](https://mlsanity.com/paper/2603.04448/) (2026)
- [SkillOS: Learning Skill Curation for Self-Evolving Agents](https://mlsanity.com/paper/2605.06614/) (2026)
- [MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild](https://mlsanity.com/paper/2603.17187/) (2026)
- [SkillGen: Verified Inference-Time Agent Skill Synthesis](https://mlsanity.com/paper/2605.10999/) (2026)
- [Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning](https://mlsanity.com/paper/2605.06130/) (2026)
- [Dynamic Dual-Granularity Skill Bank for Agentic RL](https://mlsanity.com/paper/2603.28716/) (2026)
- [Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents](https://mlsanity.com/paper/2605.30621/) (2026)
- [SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision](https://mlsanity.com/paper/2606.01139/) (2026)
- [ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2603.16060/) (2026)
- [GraSP: Graph-Structured Skill Compositions for LLM Agents](https://mlsanity.com/paper/2604.17870/) (2026)
- [SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents](https://mlsanity.com/paper/2604.17308/) (2026)
- [AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration](https://mlsanity.com/paper/2605.20025/) (2026)
- [From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills](https://mlsanity.com/paper/2605.23899/) (2026)
- [MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution](https://mlsanity.com/paper/2607.05297/) (2026)
- [GEMS: Agent-Native Multimodal Generation with Memory and Skills](https://mlsanity.com/paper/2603.28088/) (2026)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [Terminal-World: Scaling Terminal-Agent Environments via Agent Skills](https://mlsanity.com/paper/2605.20876/) (2026)
- [A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2605.07358/) (2026)
- [SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution](https://mlsanity.com/paper/2605.18401/) (2026)
- [SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents](https://mlsanity.com/paper/2605.18693/) (2026)
- [RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback](https://mlsanity.com/paper/2603.08561/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) agents have shown stunning results in complex tasks, yet they often operate in isolation, failing to learn from past experiences. Existing memory-based methods primarily store raw trajectories, which are often redundant and noise-heavy. This prevents agents from extracting high-level, reusable behavioral patterns that are essential for generalization. In this paper, we propose SkillRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution. Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank, an adaptive retrieval strategy for general and task-specific heuristics, and a recursive evolution mechanism that allows the skill library to co-evolve with the agent's policy during reinforcement learning. These innovations significantly reduce the token footprint while enhancing reasoning utility. Experimental results on ALFWorld, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance, outperforming strong baselines over 15.3% and maintaining robustness as task complexity increases. Code is available at this https://github.com/aiming-lab/SkillRL.