# How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings

arXiv: https://arxiv.org/abs/2604.04323 · HTML: https://mlsanity.com/paper/2604.04323/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 11 of 11)

- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [Memento: Fine-tuning LLM Agents without Fine-tuning LLMs](https://mlsanity.com/paper/2508.16153/) (2025)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [Reinforcement Learning for Self-Improving Agent with Skill Library](https://mlsanity.com/paper/2512.17102/) (2025)
- [SkillRouter: Skill Routing for LLM Agents at Scale](https://mlsanity.com/paper/2603.22455/) (2026)
- [Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents](https://mlsanity.com/paper/2602.01869/) (2026)
- [DynaSaur: Large Language Agents Beyond Predefined Actions](https://mlsanity.com/paper/2411.01747/) (2024)
- [Live-Evo: Online Evolution of Agentic Memory from Continuous Feedback](https://mlsanity.com/paper/2602.02369/) (2026)
- [TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agents](https://mlsanity.com/paper/2602.02196/) (2026)
- [Learning from Online Videos at Inference Time for Computer-Use Agents](https://mlsanity.com/paper/2511.04137/) (2025)

## Built on by (most cited first, 30 of 65)

- [SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision](https://mlsanity.com/paper/2606.01139/) (2026)
- [From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills](https://mlsanity.com/paper/2605.23899/) (2026)
- [SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution](https://mlsanity.com/paper/2605.18401/) (2026)
- [SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents](https://mlsanity.com/paper/2605.18693/) (2026)
- [MMSkills: Towards Multimodal Skills for General Visual Agents](https://mlsanity.com/paper/2605.13527/) (2026)
- [More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries](https://mlsanity.com/paper/2605.24050/) (2026)
- [OpenSkill: Open-World Self-Evolution for LLM Agents](https://mlsanity.com/paper/2606.06741/) (2026)
- [SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement](https://mlsanity.com/paper/2606.10546/) (2026)
- [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://mlsanity.com/paper/2608.27454/) (2026)
- [SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories](https://mlsanity.com/paper/2606.01311/) (2026)
- [SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment](https://mlsanity.com/paper/2605.27899/) (2026)
- [Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.28424/) (2026)
- [Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses](https://mlsanity.com/paper/2608.24876/) (2026)
- [SkCC: Portable and Secure Skill Compilation for Cross-Framework LLM Agents](https://mlsanity.com/paper/2605.03353/) (2026)
- [Skill Coverage: A Test Adequacy Metric for Agent Skills](https://mlsanity.com/paper/2606.20659/) (2026)
- [Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation](https://mlsanity.com/paper/2606.23127/) (2026)
- [SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use](https://mlsanity.com/paper/2607.01874/) (2026)
- [Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing](https://mlsanity.com/paper/2606.03565/) (2026)
- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)
- [A Framework for Evaluating Agentic Skills at Scale](https://mlsanity.com/paper/2606.17819/) (2026)
- [Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds](https://mlsanity.com/paper/2608.02636/) (2026)
- [SkillLens: Adaptive Multi-Granularity Skill Reuse for Cost-Efficient LLM Agents](https://mlsanity.com/paper/2605.08386/) (2026)
- [Harnessing AtomisticSkills for Agentic Atomistic Research](https://mlsanity.com/paper/2605.24002/) (2026)
- [SkillsInjector: Dynamic Skill Context Construction for LLM Agents](https://mlsanity.com/paper/2605.29794/) (2026)
- [FederatedSkill: Federated Learning for Agentic Skill Evolution](https://mlsanity.com/paper/2606.03143/) (2026)
- [Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries](https://mlsanity.com/paper/2607.10113/) (2026)
- [SkillMAS: Skill Co-Evolution with LLM-based Multi-Agent System](https://mlsanity.com/paper/2605.09341/) (2026)
- [SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization](https://mlsanity.com/paper/2606.06079/) (2026)
- [Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval](https://mlsanity.com/paper/2606.10388/) (2026)
- [Task Decomposition-Guided Reranking for Adaptive Agent Skill Retrieval](https://mlsanity.com/paper/2607.06283/) (2026)

## Abstract (arXiv)

Agent skills, which are reusable, domain-specific knowledge artifacts, have become a popular mechanism for extending LLM-based agents, yet formally benchmarking skill usage performance remains scarce. Existing skill benchmarking efforts focus on overly idealized conditions, where LLMs are directly provided with hand-crafted, narrowly-tailored task-specific skills for each task, whereas in many realistic settings, the LLM agent may have to search for and select relevant skills on its own, and even the closest matching skills may not be well-tailored for the task. In this paper, we conduct the first comprehensive study of skill utility under progressively challenging realistic settings, where agents must retrieve skills from a large collection of 34k real-world skills and may not have access to any hand-curated skills. Our findings reveal that the benefits of skills are fragile: performance gains degrade consistently as settings become more realistic, with pass rates approaching no-skill baselines in the most challenging scenarios. To narrow this gap, we study skill refinement strategies, including query-specific and query-agnostic approaches, and we show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality. We further demonstrate the generality of retrieval and refinement on Terminal-Bench 2.0, where they improve the pass rate of Claude Opus 4.6 from 57.7% to 65.5%. Our results, consistent across multiple models, highlight both the promise and the current limitations of skills for LLM-based agents. Our code is available at https://github.com/UCSB-NLP-Chang/Skill-Usage.