# SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

arXiv: https://arxiv.org/abs/2605.18693 · HTML: https://mlsanity.com/paper/2605.18693/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 20 of 20)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [SkillClaw: Let Skills Evolve Collectively with Agentic Evolver](https://mlsanity.com/paper/2604.08377/) (2026)
- [Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale](https://mlsanity.com/paper/2603.02176/) (2026)
- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [CASCADE: Cumulative Agentic Skill Creation through Autonomous Development and Evolution](https://mlsanity.com/paper/2512.23880/) (2025)
- [ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents](https://mlsanity.com/paper/2603.20340/) (2026)

## Built on by (most cited first, 18 of 18)

- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior](https://mlsanity.com/paper/2606.11543/) (2026)
- [Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories](https://mlsanity.com/paper/2607.25560/) (2026)
- [SoftSkill: Behavioral Compression for Contextual Adaptation](https://mlsanity.com/paper/2606.20333/) (2026)
- [Progressive Agent Skill Generation via Reinforcement Learning](https://mlsanity.com/paper/2608.01678/) (2026)
- [SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown](https://mlsanity.com/paper/2610.04008/) (2026)
- [SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment](https://mlsanity.com/paper/2606.22613/) (2026)
- [SkillTrace: Multi-Trace Provenance Auditing for LLM-Agent Skill Reuse](https://mlsanity.com/paper/2608.05204/) (2026)
- [SkillAligner: Treating Retrieved Skills as Adaptable Drafts at Execution Time](https://mlsanity.com/paper/2608.06880/) (2026)
- [SkillEval: Decomposing Agent Skill Quality into Interpretable Signals](https://mlsanity.com/paper/2608.06891/) (2026)
- [SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance](https://mlsanity.com/paper/2608.09253/) (2026)
- [Workflow-to-Skill: Skill Creation via Routing-Workflow-Semantics-Attachments Decomposition](https://mlsanity.com/paper/2606.06893/) (2026)
- [Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents](https://mlsanity.com/paper/2607.27484/) (2026)
- [TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation](https://mlsanity.com/paper/2608.17588/) (2026)
- [Repo2Skill-Evo: Repository Skills Go Stale in Silence](https://mlsanity.com/paper/2608.21964/) (2026)
- [mimeo: Compiling Public Expert Corpora into Agent Skills and Testing What Transfers](https://mlsanity.com/paper/2609.00453/) (2026)
- [Grounded Skill Synthesis from Code at Scale for Agentic Intelligence](https://mlsanity.com/paper/2609.05571/) (2026)
- [The Epistemics of Agent Memory: Measuring, and Governing, the Consolidation Decision in Long-Horizon LLM Agents](https://mlsanity.com/paper/2609.33013/) (2026)

## Abstract (arXiv)

As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.