# Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

arXiv: https://arxiv.org/abs/2602.01869 · HTML: https://mlsanity.com/paper/2602.01869/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 42 of its 50 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 30 of 42)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models](https://mlsanity.com/paper/2405.14831/) (2024)
- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle](https://mlsanity.com/paper/2510.16079/) (2025)
- [MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory](https://mlsanity.com/paper/2601.03192/) (2026)
- [Memento: Fine-tuning LLM Agents without Fine-tuning LLMs](https://mlsanity.com/paper/2508.16153/) (2025)

## Built on by (most cited first, 30 of 47)

- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [Memento-Skills: Let Agents Design Agents](https://mlsanity.com/paper/2603.18743/) (2026)
- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision](https://mlsanity.com/paper/2606.01139/) (2026)
- [SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution](https://mlsanity.com/paper/2605.18401/) (2026)
- [EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents](https://mlsanity.com/paper/2605.10332/) (2026)
- [Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.10923/) (2026)
- [UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation](https://mlsanity.com/paper/2606.29502/) (2026)
- [EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer](https://mlsanity.com/paper/2607.05202/) (2026)
- [User as Code: Executable Memory for Personalized Agents](https://mlsanity.com/paper/2606.16707/) (2026)
- [SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior](https://mlsanity.com/paper/2606.11543/) (2026)
- [SkillJack: Persistent Skill Backdoors in Self-Evolving Agents](https://mlsanity.com/paper/2608.03509/) (2026)
- [PreAct: Computer-Using Agents that Get Faster on Repeated Tasks](https://mlsanity.com/paper/2606.17929/) (2026)
- [Demystifying Agent Skills: Why They Work-Until They Don't](https://mlsanity.com/paper/2608.14036/) (2026)
- [SkillPyramid: A Hierarchical Skill Consolidation Framework for Self-Evolving Agents](https://mlsanity.com/paper/2606.03692/) (2026)
- [Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents](https://mlsanity.com/paper/2606.11680/) (2026)
- [LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception](https://mlsanity.com/paper/2606.30005/) (2026)
- [SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models](https://mlsanity.com/paper/2608.10538/) (2026)
- [MEMOREPAIR: Barrier-First Cascade Repair in Agentic Memory](https://mlsanity.com/paper/2605.07242/) (2026)
- [MAP: A Map-then-Act Paradigm for Long-Horizon Interactive Agent Reasoning](https://mlsanity.com/paper/2605.13037/) (2026)
- [R^2-Mem: Reflective Experience for Memory Search](https://mlsanity.com/paper/2605.13486/) (2026)
- [Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning](https://mlsanity.com/paper/2606.00510/) (2026)
- [Metis: Bridging Text and Code Memory for Self-Evolving Agents](https://mlsanity.com/paper/2606.24151/) (2026)
- [Progressive Agent Skill Generation via Reinforcement Learning](https://mlsanity.com/paper/2608.01678/) (2026)
- [From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents](https://mlsanity.com/paper/2608.09168/) (2026)
- [SkillBrew: Multi-Objective Curation of Skill Banks for LLM Agents](https://mlsanity.com/paper/2605.29440/) (2026)
- [SkillZip: Contract-Preserving Graph Compression for Scalable Agent Skill Libraries](https://mlsanity.com/paper/2608.05604/) (2026)
- [When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents](https://mlsanity.com/paper/2608.22339/) (2026)
- [Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design](https://mlsanity.com/paper/2609.22086/) (2026)
- [ERRAND: Budgeted Maintenance of Agent Memory](https://mlsanity.com/paper/2609.29545/) (2026)

## Abstract (arXiv)

LLM-driven agents excel at sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and instability. To bridge this gap, we propose Skill-Pro, a framework enabling agents to autonomously learn reusable procedural skills from interaction experiences without parameter updates. By formalizing a Skill-MDP, Skill-Pro transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, Skill-Pro sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that Skill-Pro achieves superior reuse rates and significant gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how Skill-Pro transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.