# SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

arXiv: https://arxiv.org/abs/2603.15401 · HTML: https://mlsanity.com/paper/2603.15401/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 10 of 10)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward](https://mlsanity.com/paper/2602.12430/) (2026)
- [Help or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models](https://mlsanity.com/paper/2508.12566/) (2025)

## Built on by (most cited first, 30 of 82)

- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization](https://mlsanity.com/paper/2604.02268/) (2026)
- [SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents](https://mlsanity.com/paper/2605.05726/) (2026)
- [SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision](https://mlsanity.com/paper/2606.01139/) (2026)
- [SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents](https://mlsanity.com/paper/2604.17308/) (2026)
- [From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills](https://mlsanity.com/paper/2605.23899/) (2026)
- [SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents](https://mlsanity.com/paper/2605.18693/) (2026)
- [SkVM: Revisiting Language VM for Skills across Heterogenous LLMs and Harnesses](https://mlsanity.com/paper/2604.03088/) (2026)
- [SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering](https://mlsanity.com/paper/2604.09297/) (2026)
- [CODESKILL: Learning Self-Evolving Skills for Coding Agents](https://mlsanity.com/paper/2605.25430/) (2026)
- [Counterfactual Trace Auditing of LLM Agent Skills](https://mlsanity.com/paper/2605.11946/) (2026)
- [Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents](https://mlsanity.com/paper/2606.16769/) (2026)
- [Agent Skill Evaluation and Evolution: Frameworks and Benchmarks](https://mlsanity.com/paper/2606.11435/) (2026)
- [SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History](https://mlsanity.com/paper/2606.08671/) (2026)
- [Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds](https://mlsanity.com/paper/2608.02636/) (2026)
- [SkillGuard: A Permission-Centric Framework for Agent Skill Security](https://mlsanity.com/paper/2606.03024/) (2026)
- [SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior](https://mlsanity.com/paper/2606.11543/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills](https://mlsanity.com/paper/2607.01456/) (2026)
- [Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries](https://mlsanity.com/paper/2607.10113/) (2026)
- [ASDA: Automated Skill Distillation and Adaptation for Financial Reasoning](https://mlsanity.com/paper/2603.16112/) (2026)
- [SkillMAS: Skill Co-Evolution with LLM-based Multi-Agent System](https://mlsanity.com/paper/2605.09341/) (2026)
- [SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills](https://mlsanity.com/paper/2605.24117/) (2026)
- [S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agents](https://mlsanity.com/paper/2606.15367/) (2026)
- [GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents](https://mlsanity.com/paper/2606.24551/) (2026)
- [Demystifying Agent Skills: Why They Work-Until They Don't](https://mlsanity.com/paper/2608.14036/) (2026)
- [ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation](https://mlsanity.com/paper/2604.23853/) (2026)
- [Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval](https://mlsanity.com/paper/2606.10388/) (2026)
- [Task Decomposition-Guided Reranking for Adaptive Agent Skill Retrieval](https://mlsanity.com/paper/2607.06283/) (2026)
- [Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Libraries](https://mlsanity.com/paper/2605.10990/) (2026)

## Abstract (arXiv)

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.