mlsanityData to 7 Oct 2026

Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI

arXiv 2607.02873 ↗First posted 3 Jul 2026cs.SE cs.AI cs.CRcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language model agents driving security tool suites over the Model Context Protocol are increasingly common. Yet the factors that bound their capability remain poorly characterized: how much depends on the model versus the client that drives it, whether constraining the agent to the orchestrator's own tools helps, and where capability is limited by reasoning rather than by missing tools. Using HexStrikeAI, an open-source orchestrator that exposes 150+ tools, as a testbed, we follow a methodology that evaluates the system, diagnoses its failures, and applies targeted improvements. We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations (774 trials). We then apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials. The diagnosis isolates the driving client as a first-order factor for a fixed model (a 2.1 * gap between two DeepSeek clients) and a monotonic difficulty gradient, with the largest gains in the mid tier. The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly (paired McNemar p < 0.001, non-overlapping 95% confidence intervals). The residual failures are reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study finds single-run verdicts reproducible (17/20 unanimous). We discuss what the results imply for how such orchestrators should be evaluated, and we are explicit about the limits: the study uses a single benchmark, the fixes were tuned on the same challenges they were evaluated on, and the client effect is demonstrated for one model only, so its generality to other models remains a hypothesis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 21 of its 26 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

21 of 26 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity19 May
  2. 2026Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges21 Apr
  3. 2026What Makes a Good LLM Agent for Real-world Penetration Testing?19 Feb
  4. 2025Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark5 Aug
  5. 2025Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing14 Jul
  6. 2025CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution21 May
  7. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities24 Sep

Other references · 14

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  6. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  7. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 44
  8. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 25
  9. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 13
  10. 2024Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvementscited by 11
  11. 2026How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelinescited by 5
  12. 2026Towards Cybersecurity SuperIntelligence (CSI): What's the best harness for cybersecurity?cited by 2
2 more
  1. 2026How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistencycited by 1
  2. 2026Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Predefined Sinks: Security-Aware Dependency Analysis for LLM Agents
  2. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  3. 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms
  4. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
  5. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  6. 2026Autonomous LLM Agents & CTFs: A Second Look
  7. 2026LLM-Based Agents for Software and Systems Security: Approaches, Applications, and Assessment
  8. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents