# Skill Coverage: A Test Adequacy Metric for Agent Skills arXiv: https://arxiv.org/abs/2606.20659 ยท HTML: https://mlsanity.com/paper/2606.20659/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use. - Builds on 18 of its 37 references that are in the map. - First posted on arXiv 9 Jun 2026. ## Placed in - LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/) ## Builds on (most cited first, 18 of 18) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026) - [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024) - [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024) - [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026) - [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023) - [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023) - [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025) - [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025) - [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026) - [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023) - [Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality](https://mlsanity.com/paper/2602.08004/) (2026) - [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026) - [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026) - [CASCADE: Cumulative Agentic Skill Creation through Autonomous Development and Evolution](https://mlsanity.com/paper/2512.23880/) (2025) - [A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2605.07358/) (2026) - [OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents](https://mlsanity.com/paper/2605.23657/) (2026) ## Built on by (most cited first, 8 of 8) - [Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories](https://mlsanity.com/paper/2607.25560/) (2026) - [SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment](https://mlsanity.com/paper/2608.07639/) (2026) - [Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents](https://mlsanity.com/paper/2608.08793/) (2026) - [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026) - [The Work Behind Delegation: A Framework for Supervising AI Coding Agents](https://mlsanity.com/paper/2609.24234/) (2026) - [How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure](https://mlsanity.com/paper/2609.30074/) (2026) - [When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use](https://mlsanity.com/paper/2609.32274/) (2026) - [Organising Trajectory Evidence for Language-Model Agent Assurance: Fragments, Methods, and the Residual](https://mlsanity.com/paper/2610.04710/) (2026) ## Abstract (arXiv) Agent skills encode reusable procedural knowledge for large language model (LLM) agents, and existing benchmarks show that such skills can improve task-level performance. However, a task outcome does not reveal which parts of a reusable skill were exercised, nor whether the agent followed the relevant skill instructions when those parts were exercised. This gap makes it unclear whether a skill has been adequately tested, or whether observed task failures provide actionable evidence for improving agent skill effectiveness. To fill this gap, we introduce skill coverage, a trajectory-based test-adequacy metric for reusable agent skills. Our framework extracts skill behavior constraints from each skill, translating natural-language skill instructions into semi-structured constraints that specify the expected agent behavior under particular conditions. It then determines whether each constraint is covered by an agent trajectory and, for covered constraints, assigns a Pass or Fail verdict according to the agent behavior. We apply this framework to SkillsBench. The results show that agent trajectories on the benchmark leaderboard cover only 38.66 to 45.51% of the extracted skill behavior constraints on average. We then use Fail verdicts to strengthen the corresponding skill content only by emphasizing the original instructions that the agent failed to follow, and run the same tasks with the strengthened skills. This emphasis yields an average 16.0% recovery rate of the failed tasks across the five agent-model rows. These results show that skill coverage is both a test-adequacy metric and a fine-grained signal for observing skill-use behavior. In failed tasks, failed constraint labels provide actionable evidence for improving agent skill effectiveness. A project website accompanies the paper.