# Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

arXiv: https://arxiv.org/abs/2402.03244 · HTML: https://mlsanity.com/paper/2402.03244/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization](https://mlsanity.com/paper/2310.10134/) (2023)
- [Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling](https://mlsanity.com/paper/2301.12050/) (2023)

## Built on by (most cited first, 26 of 26)

- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training](https://mlsanity.com/paper/2501.11425/) (2025)
- [SkillClaw: Let Skills Evolve Collectively with Agentic Evolver](https://mlsanity.com/paper/2604.08377/) (2026)
- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought](https://mlsanity.com/paper/2406.14596/) (2024)
- [SelfGoal: Your Language Agents Already Know How to Achieve High-level Goals](https://mlsanity.com/paper/2406.04784/) (2024)
- [The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement](https://mlsanity.com/paper/2503.16024/) (2025)
- [Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search](https://mlsanity.com/paper/2408.10635/) (2024)
- [AgentKit: Structured LLM Reasoning with Dynamic Graphs](https://mlsanity.com/paper/2404.11483/) (2024)
- [When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents](https://mlsanity.com/paper/2604.27003/) (2026)
- [Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents](https://mlsanity.com/paper/2606.16769/) (2026)
- [Enhancing Agent Learning through World Dynamics Modeling](https://mlsanity.com/paper/2407.17695/) (2024)
- [Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2506.23127/) (2025)
- [H2R: Hierarchical Hindsight Reflection for Multi-Task LLM Agents](https://mlsanity.com/paper/2509.12810/) (2025)
- [ABSTRAL: Automatic Design of Multi-Agent Systems Through Iterative Refinement and Topology Optimization](https://mlsanity.com/paper/2603.22791/) (2026)
- [MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learning](https://mlsanity.com/paper/2411.12977/) (2024)
- [SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding](https://mlsanity.com/paper/2603.09036/) (2026)
- [Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents](https://mlsanity.com/paper/2608.20274/) (2026)
- [SkillShapley: Boundary-Adaptive Shapley Valuation for Skill Step Attribution in LLM Agents](https://mlsanity.com/paper/2608.13173/) (2026)
- [CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task](https://mlsanity.com/paper/2509.24422/) (2025)
- [Towards Scalable Customization and Deployment of Multi-Agent Systems for Enterprise Applications](https://mlsanity.com/paper/2606.18502/) (2026)
- [TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation](https://mlsanity.com/paper/2608.17588/) (2026)
- [CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolution](https://mlsanity.com/paper/2609.04865/) (2026)
- [SEER: Self-Evolving Event Reasoning and Retrieval for Time Series Forecasting](https://mlsanity.com/paper/2610.04109/) (2026)
- [SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection](https://mlsanity.com/paper/2610.18807/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have recently been used for sequential decision making in interactive environments. However, leveraging environment reward signals for continual LLM actor improvement is not straightforward. We propose Skill Set Optimization (SSO) for improving LLM actor performance through constructing and refining sets of transferable skills. SSO constructs skills by extracting common subtrajectories with high rewards and generating subgoals and instructions to represent each skill. These skills are provided to the LLM actor in-context to reinforce behaviors with high rewards. Then, SSO further refines the skill set by pruning skills that do not continue to result in high rewards. We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement. SSO outperforms baselines by 40% in our custom NetHack task and outperforms the previous state-of-the-art in ScienceWorld by 35%.