mlsanityData to 14 Oct 2026

When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use

arXiv 2609.32274 ↗First posted 26 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Agent skills are expected to improve task performance. Yet we find that they often provide no benefit, and can even hurt performance while incurring additional token costs. Can we predict whether a skill will help before the agent acts? We introduce SkillDelta, a framework for predicting task-conditional skill gains from paired executions of the same agent with and without the skill. A local predictor transfers these historical gains to new tasks without retraining the agent. Under explicit transfer assumptions, our analysis links support coverage, representation mismatch, and execution noise to prediction error and decision regret. Across five benchmarks and three target agents, paired history improves observed-gain ranking over skill-assisted outcomes alone in 12 of 15 settings. At matched expected skill-use rates, SkillDelta improves success over random activation in all 15 settings, with an average absolute gain of 4.3%. Most of this advantage comes from allocation across task groups. Evidence for additional within-group selection value is strongest on ToolQA and weaker elsewhere. Code is available at https://github.com/TankTechnology/skilldelta.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

28 of 37 references.

References · 28

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2018Computational Optimal Transportcited by 780
  4. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  5. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  6. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 478
  7. 2015Optimal Transport for Domain Adaptationcited by 463
  8. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  9. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 332
  10. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
  11. 2016Generalized Random Forestscited by 312
  12. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
16 more
  1. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  2. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 148
  3. 2016Theoretical Analysis of Domain Adaptation with Optimal Transportcited by 105
  4. 2026SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?cited by 82
  5. 2018Offline Multi-Action Policy Learning: Generalization and Optimizationcited by 65
  6. 2026How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settingscited by 65
  7. 2026SkillRouter: Skill Routing for LLM Agents at Scalecited by 61
  8. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 47
  9. 2026Skill Retrieval Augmentation for Agentic AIcited by 44
  10. 2025CARROT: A Cost Aware Rate Optimal Routercited by 39
  11. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
  12. 2021A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modelingcited by 12
  13. 2026SkillTester: Benchmarking Utility and Security of Agent Skillscited by 10
  14. 2026Skill Coverage: A Test Adequacy Metric for Agent Skillscited by 8
  15. 2024Can We Predict Performance of Large Models across Vision-Language Tasks?cited by 6
  16. 2025Domain Adaptation and Entanglement: an Optimal Transport Perspectivecited by 6

Built on by

Not cited yet.