# SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

arXiv: https://arxiv.org/abs/2605.05726 · HTML: https://mlsanity.com/paper/2605.05726/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 30 of 31)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward](https://mlsanity.com/paper/2602.12430/) (2026)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale](https://mlsanity.com/paper/2603.02176/) (2026)
- [Memento-Skills: Let Agents Design Agents](https://mlsanity.com/paper/2603.18743/) (2026)
- [Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models](https://mlsanity.com/paper/2405.05374/) (2024)
- [SkillRouter: Skill Routing for LLM Agents at Scale](https://mlsanity.com/paper/2603.22455/) (2026)
- [Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models](https://mlsanity.com/paper/2503.01763/) (2025)
- [MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild](https://mlsanity.com/paper/2603.17187/) (2026)
- [KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model](https://mlsanity.com/paper/2506.20923/) (2025)
- [Towards Completeness-Oriented Tool Retrieval for Large Language Models](https://mlsanity.com/paper/2405.16089/) (2024)
- [WebXSkill: Skill Learning for Autonomous Web Agents](https://mlsanity.com/paper/2604.13318/) (2026)
- [SkillFlow: Scalable and Efficient Agent Skill Retrieval System](https://mlsanity.com/paper/2504.06188/) (2025)
- [jina-embeddings-v5-text: Task-Targeted Embedding Distillation](https://mlsanity.com/paper/2602.15547/) (2026)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)

## Built on by (most cited first, 30 of 39)

- [MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation](https://mlsanity.com/paper/2605.27366/) (2026)
- [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://mlsanity.com/paper/2608.27454/) (2026)
- [LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents](https://mlsanity.com/paper/2606.06087/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment](https://mlsanity.com/paper/2605.27899/) (2026)
- [Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing](https://mlsanity.com/paper/2606.03565/) (2026)
- [Skill-as-Pseudocode: Refactoring Skill Libraries to Pseudocode for LLM Agents](https://mlsanity.com/paper/2605.27955/) (2026)
- [Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval](https://mlsanity.com/paper/2606.10388/) (2026)
- [SkillSight: Calibrating Generic Content Bias for Skill Retrieval](https://mlsanity.com/paper/2607.18785/) (2026)
- [Field-Aware Agent Skill Retrieval](https://mlsanity.com/paper/2608.02880/) (2026)
- [Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security](https://mlsanity.com/paper/2606.11671/) (2026)
- [SKILL-KD: Contrastive Skill Distillation for LLM Agents](https://mlsanity.com/paper/2607.28048/) (2026)
- [Beyond Task-Only Matching: Personalized Skill Routing with Counterfactual Evaluation](https://mlsanity.com/paper/2608.28241/) (2026)
- [Imagine Before Retrieval: Prospective Skill Retrieval for LLM Agents](https://mlsanity.com/paper/2609.01642/) (2026)
- [SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents](https://mlsanity.com/paper/2607.15557/) (2026)
- [Behavioral Skill Reconstruction: Reconstructing Hidden Functionality from LLM Agent Skills](https://mlsanity.com/paper/2608.04192/) (2026)
- [SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests](https://mlsanity.com/paper/2608.08640/) (2026)
- [From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents](https://mlsanity.com/paper/2608.09168/) (2026)
- [Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval](https://mlsanity.com/paper/2608.16071/) (2026)
- [Optimal Skill Selection for LLM Agents with Provable Bicriteria Guarantees](https://mlsanity.com/paper/2608.19993/) (2026)
- [Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models](https://mlsanity.com/paper/2606.12203/) (2026)
- [SkillTrace: Traversing a Query-Skill Graph for Composable LLM Agents](https://mlsanity.com/paper/2608.02356/) (2026)
- [SkillZip: Contract-Preserving Graph Compression for Scalable Agent Skill Libraries](https://mlsanity.com/paper/2608.05604/) (2026)
- [Learning Globally Reusable Skills for Coding Agents](https://mlsanity.com/paper/2608.06153/) (2026)
- [SkillAligner: Treating Retrieved Skills as Adaptable Drafts at Execution Time](https://mlsanity.com/paper/2608.06880/) (2026)
- [Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories](https://mlsanity.com/paper/2608.10319/) (2026)
- [SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents](https://mlsanity.com/paper/2608.18852/) (2026)
- [COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization](https://mlsanity.com/paper/2609.11682/) (2026)
- [The Router Within: Eliciting Native Skill Routing from a Frozen LLM](https://mlsanity.com/paper/2609.15982/) (2026)
- [SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale](https://mlsanity.com/paper/2609.38822/) (2026)

## Abstract (arXiv)

As LLM agents are increasingly deployed with large libraries of reusable skills, selecting the right skill for a user request has become a critical systems challenge. In small libraries, users may invoke skills explicitly by name, but this assumption breaks down as skill ecosystems grow under tight context and latency budgets. Despite its practical importance, skill retrieval remains underexplored, with limited benchmarks and little understanding of retrieval behavior on realistic skill libraries. To address this gap, we introduce SkillRet, a large-scale benchmark for skill retrieval in LLM agents. SkillRet contains 16,129 public agent skills, organized with structured semantic tags and a two-level taxonomy spanning 6 major categories and 18 sub-categories. It provides 63,259 training samples and 4,392 evaluation queries with disjoint skill pools, enabling both benchmarking and retrieval-oriented training. Across a diverse set of retrievers, we find that skill retrieval remains far from solved: off-the-shelf models struggle on realistic large-scale skill libraries, and prior skill-retrieval models still leave substantial headroom. Task-specific fine-tuning on SkillRet improves NDCG@10 by 12.9 points over the strongest prior retriever and by 16.2 points over the strongest off-the-shelf retriever. Our analysis further suggests that these gains arise because fine-tuned models better focus on the small skill-relevant signals within long and noisy queries. These results establish SkillRet as a strong benchmark and foundation for future research on retrieval in large-scale agent systems. We publicly release the benchmark (https://huggingface.co/datasets/ThakiCloud/SKILLRET), code (https://github.com/ThakiCloud/SKILLRET), and model checkpoints (0.6B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-0.6B; 8B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-8B).