# SkillEval: Decomposing Agent Skill Quality into Interpretable Signals

arXiv: https://arxiv.org/abs/2608.06891 · HTML: https://mlsanity.com/paper/2608.06891/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality](https://mlsanity.com/paper/2602.08004/) (2026)
- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026)
- [Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation](https://mlsanity.com/paper/2505.16222/) (2025)
- [SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents](https://mlsanity.com/paper/2605.18693/) (2026)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)
- [From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills](https://mlsanity.com/paper/2604.24026/) (2026)
- [Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry](https://mlsanity.com/paper/2605.11418/) (2026)
- [Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations](https://mlsanity.com/paper/2601.09833/) (2026)
- [Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry](https://mlsanity.com/paper/2601.22588/) (2026)
- [SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use](https://mlsanity.com/paper/2607.01874/) (2026)
- [A Framework for Evaluating Agentic Skills at Scale](https://mlsanity.com/paper/2606.17819/) (2026)
- [SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills](https://mlsanity.com/paper/2605.24117/) (2026)
- [OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents](https://mlsanity.com/paper/2605.23657/) (2026)
- [Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments](https://mlsanity.com/paper/2602.16653/) (2026)

## Built on by (most cited first, 1 of 1)

- [Rep2Skill: Representation-Guided Skill Self-Evolution for LLM Agents](https://mlsanity.com/paper/2609.39149/) (2026)

## Abstract (arXiv)

Agent skills provide reusable knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality based on improvements in agent performance on specific tasks. However, a reusable skill may apply across multiple scenarios. Evaluation on a single downstream task mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspects need improvement. We find that general properties of the textttSKILL.md document play an important role in skill quality. To evaluate these properties, we propose SkillEval, an interpretable framework for document-level skill evaluation. SkillEval produces interpretable scores for each property using a fixed, inspectable scoring direction. It further measures and reduces the influence of unrelated document features, so each score more specifically captures its intended semantic property. We use SkillEval to evaluate skills in controlled quality tests, showing that it reliably distinguishes skills of different quality. Its scores closely reflect downstream task performance, providing an early indication of skill effectiveness. We further explore SkillEval for diagnosing document weaknesses and guiding targeted revisions that improve the targeted properties and increase downstream task pass rates.