mlsanityData to 14 Oct 2026

Reinforcement Learning for Self-Improving Agent with Skill Library

arXiv 2512.17102 ↗First posted 18 Dec 2025cs.AIcited by 100

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in complex reasoning and multi-turn interactions but struggle to continuously improve and adapt when deployed in new environments. One promising approach is implementing skill libraries that allow agents to learn, validate, and apply new skills. However, current skill library approaches rely primarily on LLM prompting, making consistent skill library implementation challenging. To overcome these challenges, we propose a Reinforcement Learning (RL)-based approach to enhance agents' self-improvement capabilities with a skill library. Specifically, we introduce Skill Augmented GRPO for self-Evolution (SAGE), a novel RL framework that systematically incorporates skills into learning. The framework's key component, Sequential Rollout, iteratively deploys agents across a chain of similar tasks for each rollout. As agents navigate through the task chain, skills generated from previous tasks accumulate in the library and become available for subsequent tasks. Additionally, the framework enhances skill generation and utilization through a Skill-integrated Reward that complements the original outcome-based rewards. Experimental results on AppWorld demonstrate that SAGE, when applied to supervised-finetuned model with expert experience, achieves 8.9% higher Scenario Goal Completion while requiring 26% fewer interaction steps and generating 59% fewer tokens, substantially outperforming existing approaches in both accuracy and efficiency.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 31 of its 38 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

31 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks19 Mar
  2. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents13 Aug

Other references · 29

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  10. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
17 more
  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  2. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  4. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  5. 2023Large Language Models as Tool Makerscited by 303
  6. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  7. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  8. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  10. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  11. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  12. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  13. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  14. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  15. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  16. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  17. 2025A Survey of Model Architectures in Information Retrievalcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 99.

100 papers.

Sort:

By area:

  1. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 148
  2. 2026CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verificationcited by 128
  3. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
  4. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  5. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 76
  6. 2026How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settingscited by 65
  7. 2026SkillNet: Create, Evaluate, and Connect AI Skillscited by 53
  8. 2026SkillOS: Learning Skill Curation for Self-Evolving Agentscited by 49
  9. 2026SKILLFOUNDRY: Building Self-Evolving Agent Skill Libraries from Heterogeneous Scientific Resourcescited by 41
  10. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  11. 2026ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learningcited by 29
  12. 2026From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skillscited by 22
38 more
  1. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  2. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 18
  3. 2026SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystemscited by 18
  4. 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolutioncited by 18
  5. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Taskscited by 15
  6. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 15
  7. 2026CODESKILL: Learning Self-Evolving Skills for Coding Agentscited by 12
  8. 2026OpenSkill: Open-World Self-Evolution for LLM Agentscited by 12
  9. 2026SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinementcited by 12
  10. 2026SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditingcited by 11
  11. 2026SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topologycited by 9
  12. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  13. 2026Harnessing LLM Agents with Skill Programscited by 9
  14. 2025Audited Skill-Graph Self-Improvement for Agentic LLMs via Verifiable Rewards, Experience Synthesis, and Continual Memorycited by 8
  15. 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agentscited by 8
  16. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  17. 2026SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Usecited by 8
  18. 2026Evolving Programmatic Skill Networkscited by 7
  19. 2026Co-Evolving Skill Generation and Policy Optimizationcited by 7
  20. 2026Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remembercited by 7
  21. 2026SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestrationcited by 6
  22. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 6
  23. 2026Not All Skills Help: Measuring and Repairing Agent Knowledgecited by 6
  24. 2026Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Librariescited by 6
  25. 2026Self-Improving World Modelling with Latent Actionscited by 5
  26. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  27. 2026From Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agentscited by 5
  28. 2026SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolutioncited by 5
  29. 2026When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination in LLM Agentscited by 5
  30. 2026Bilevel Optimization of Agent Skills via Monte Carlo Tree Searchcited by 4
  31. 2026Task Decomposition-Guided Reranking for Adaptive Agent Skill Retrievalcited by 4
  32. 2026Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectoriescited by 4
  33. 2026Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agentscited by 3
  34. 2026Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agentscited by 3
  35. 2026Understanding Rollout Error in Graph World Modelscited by 3
  36. 2026SKT: Skill-Use Training at Scale via Verified Synthetic Data Generationcited by 3
  37. 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 3
  38. 2026EVOMAL: Self-Poisoning in Self-Evolving Coding Agentscited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
  2. 2026SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles
  3. 2026Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO
  4. 2026SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents
  5. 2026SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization
  6. 2026Search2Skill: Skill Distillation Beyond Knowledge Boundaries Via Rubric-Based Reinforcement Learning
  7. 2026SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs
  8. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents