# Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval

arXiv: https://arxiv.org/abs/2606.10388 · HTML: https://mlsanity.com/paper/2606.10388/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [SkillRouter: Skill Routing for LLM Agents at Scale](https://mlsanity.com/paper/2603.22455/) (2026)
- [Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents](https://mlsanity.com/paper/2405.16533/) (2024)
- [Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills](https://mlsanity.com/paper/2604.05333/) (2026)
- [Skill Retrieval Augmentation for Agentic AI](https://mlsanity.com/paper/2604.24594/) (2026)
- [ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval](https://mlsanity.com/paper/2403.06551/) (2024)
- [SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents](https://mlsanity.com/paper/2605.05726/) (2026)
- [MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills](https://mlsanity.com/paper/2606.07131/) (2026)
- [AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills](https://mlsanity.com/paper/2605.13940/) (2026)
- [Group of Skills: Group-Structured Skill Retrieval for Agent Skill Libraries](https://mlsanity.com/paper/2605.06978/) (2026)
- [SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution](https://mlsanity.com/paper/2605.10114/) (2026)
- [Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing](https://mlsanity.com/paper/2606.03565/) (2026)
- [SkillGuard: A Permission-Centric Framework for Agent Skill Security](https://mlsanity.com/paper/2606.03024/) (2026)
- [OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents](https://mlsanity.com/paper/2605.23657/) (2026)

## Built on by (most cited first, 4 of 4)

- [From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents](https://mlsanity.com/paper/2608.09168/) (2026)
- [SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents](https://mlsanity.com/paper/2608.18852/) (2026)
- [What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review](https://mlsanity.com/paper/2609.33153/) (2026)
- [SkillContrast: Difference-Guided Text Selection for Agent Skill Reranking](https://mlsanity.com/paper/2610.11650/) (2026)

## Abstract (arXiv)

A skill can match a task's topic while conflicting with its resource, procedure, or output requirements. We study this as same-capability risk-exposure retrieval and introduce SameCapRisk-Bench: 890 units and 1,314 query cases across five mechanisms and twelve conflict types. Each unit pairs a skill that meets a query requirement with a same-capability skill that violates it, with evidence for that distinction. The evaluation tests source-task requirements and two kinds of paired queries that reverse which skill fits: changing the requested evidence role or exact output interface while holding the skills fixed. To capture both retrieval success and conflicting exposure, Recall tracks helpful hits, harmful sibling rate (HSR) tracks exposure of the conflicting sibling, and CleanHit requires a helpful hit without that exposure. Four public skill retrievers expose conflicting siblings at HSR@3 of 0.737-0.881 on source-task contracts, compared with 0.099-0.12 on controlled source-role queries. Across the fixed mixture of 1,235 held-out queries, their Recall@3 is 0.903-0.944 and HSR@3 is 0.344-0.393. The conflicting sibling ranks first on 24.8-29.2% of source-task queries for these four retrievers. We also examine what the reranker receives: truncating long skills can remove the passages where the two skills' contracts differ. With the same BGE top-20 candidates, increasing the reranker's input budget from 512 to 4,096 tokens reduces source-task sibling-first errors by 8.27 pp, with an uncertain CleanHit gain.