# Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills

arXiv: https://arxiv.org/abs/2608.20614 · HTML: https://mlsanity.com/paper/2608.20614/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 9 of 9)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration](https://mlsanity.com/paper/2603.03823/) (2026)
- [SkillTester: Benchmarking Utility and Security of Agent Skills](https://mlsanity.com/paper/2603.28815/) (2026)
- [A Framework for Evaluating Agentic Skills at Scale](https://mlsanity.com/paper/2606.17819/) (2026)
- [GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows](https://mlsanity.com/paper/2604.15715/) (2026)
- [AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows](https://mlsanity.com/paper/2607.16345/) (2026)

## Built on by (most cited first, 2 of 2)

- [Scanning the Harness: A Repository Study of Configuration Exposures in AI Coding Agents](https://mlsanity.com/paper/2609.07360/) (2026)
- [Mining Agent Skills from Production Traces](https://mlsanity.com/paper/2610.05777/) (2026)

## Abstract (arXiv)

Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman ρ = 0.14). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.