# A Framework for Evaluating Agentic Skills at Scale

arXiv: https://arxiv.org/abs/2606.17819 · HTML: https://mlsanity.com/paper/2606.17819/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 9 of 9)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale](https://mlsanity.com/paper/2602.23866/) (2026)
- [Agentic Rubrics as Contextual Verifiers for SWE Agents](https://mlsanity.com/paper/2601.04171/) (2026)

## Built on by (most cited first, 7 of 7)

- [Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills](https://mlsanity.com/paper/2608.20614/) (2026)
- [Scientific Agent Skills: A Library of Procedural Knowledge for Research Agents](https://mlsanity.com/paper/2609.00065/) (2026)
- [SkillEval: Decomposing Agent Skill Quality into Interpretable Signals](https://mlsanity.com/paper/2608.06891/) (2026)
- [COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization](https://mlsanity.com/paper/2609.11682/) (2026)
- [MCRI: A Four-Dimensional Framework for Analyzing and Evaluating Agent Skills](https://mlsanity.com/paper/2610.01506/) (2026)
- [One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents](https://mlsanity.com/paper/2610.11647/) (2026)
- [SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications](https://mlsanity.com/paper/2610.14918/) (2026)

## Abstract (arXiv)

Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.