# When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity

arXiv: https://arxiv.org/abs/2605.20023 · HTML: https://mlsanity.com/paper/2605.20023/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 19 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 5 of 5)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](https://mlsanity.com/paper/2305.13971/) (2023)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving](https://mlsanity.com/paper/2603.22577/) (2026)

## Built on by (most cited first, 4 of 4)

- [Scientific Agent Skills: A Library of Procedural Knowledge for Research Agents](https://mlsanity.com/paper/2609.00065/) (2026)
- [Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI](https://mlsanity.com/paper/2607.02873/) (2026)
- [Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security](https://mlsanity.com/paper/2608.29596/) (2026)
- [Memory as Middleware for Self-Improving AI Agents](https://mlsanity.com/paper/2609.32091/) (2026)

## Abstract (arXiv)

Agent Skills, structured packages of procedural knowledge loaded into an LLM agent at inference time, are widely reported to improve task pass rates by an average of 16.2~percentage points across diverse domains. Yet the same benchmarks show wide variance, with 16 of 84 tasks suffering negative deltas when Skills are introduced. The community has not yet articulated a clean mechanism for when Skills help and when they are merely redundant overhead. We re-analyze a recently published 180-run controlled study of an MCP-grounded autonomous Capture-the-Flag (CTF) agent under four documentation conditions of increasing richness (591, 12865, 17253, and 36001 tokens) and show that these conditions correspond almost exactly to a No-Skills, Experiential-Skills, Curated-Skills, and Comprehensive-Skills ablation. In offensive cybersecurity, a domain not deeply covered by existing Skills benchmarks, the marginal benefit of Skills collapses. The spread between the no-Skills and full-Skills conditions is only 8.9~pp (p = 0.71, chi2; p = 0.25, Cochran--Armitage trend test; five of six pairwise Cohen's h values fall below the 0.2 small-effect threshold). We argue that the missing variable is environment-feedback bandwidth. When an agent's tool layer returns strict, schema-validated, low-latency observations, the environment itself supplies the procedural correction signal that Skills are normally needed to provide. As a result, the marginal benefit of curated Skills diminishes substantially, and, in some cases (e.g., our timing side-channel setting), actively degrades performance. We articulate a falsifiable hypothesis, sketch its design implications for compound AI systems, and will release the reanalysis pipeline to support replication.