mlsanityData to 14 Oct 2026

From Capabilities to Performance: Evaluating Key Functional Properties of LLM Architectures in Penetration Testing

arXiv 2509.14289 ↗First posted 16 Sep 2025cs.AI cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used to automate or augment penetration testing, but their effectiveness and reliability across attack phases remain unclear. We present a comprehensive evaluation of multiple LLM-based agents, from single-agent to modular designs, across realistic penetration testing scenarios, measuring empirical performance and recurring failure patterns. We also isolate the impact of five core functional capabilities via targeted augmentations: Global Context Memory (GCM), Inter-Agent Messaging (IAM), Context-Conditioned Invocation (CCI), Adaptive Planning (AP), and Real-Time Monitoring (RTM). These interventions support, respectively: (i) context coherence and retention, (ii) inter-component coordination and state management, (iii) tool use accuracy and selective execution, (iv) multi-step strategic planning, error detection, and recovery, and (v) real-time dynamic responsiveness. Our results show that while some architectures natively exhibit subsets of these properties, targeted augmentations substantially improve modular agent performance, especially in complex, multi-step, and real-time penetration testing tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 70 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

30 of 70 references.

References · 30

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  5. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  6. 2025A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)cited by 101
  7. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 97
  8. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 96
  9. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 95
  10. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 37
  11. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  12. 2024A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directionscited by 33
18 more
  1. 2024INMS: Memory Sharing for Large Language Model based Agentscited by 31
  2. 2024PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediationcited by 30
  3. 2023Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questionscited by 23
  4. 2024Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilitiescited by 21
  5. 2023RatGPT: Turning online LLMs into Proxies for Malware Attackscited by 18
  6. 2023Out of the Cage: How Stochastic Parrots Win in Cyber Security Environmentscited by 18
  7. 2024LLMCloudHunter: Harnessing LLMs for Automated Extraction of Detection Rules from Cloud-Based CTIcited by 18
  8. 2023LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testingcited by 17
  9. 2022Multi-Agent Coordination via Multi-Level Communicationcited by 14
  10. 2023From Chatbots to PhishBots? -- Preventing Phishing scams created using ChatGPT, Google Bard and Claudecited by 13
  11. 2024Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilitiescited by 12
  12. 2022Discovering Exfiltration Paths Using Reinforcement Learning with Attack Graphscited by 8
  13. 2024A Survey of Large Language Models in Cybersecuritycited by 8
  14. 2024Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applicationscited by 8
  15. 2025Position: AI Safety Must Embrace an Antifragile Perspectivecited by 5
  16. 2022Exposing Surveillance Detection Routes via Reinforcement Learning, Attack Graphs, and Cyber Terraincited by 3
  17. 2023Fast and Adaptive Multi-agent Planning under Collaborative Temporal Logic Tasks via Poset Productscited by 1
  18. 2024Leveraging Reinforcement Learning in Red Teaming for Advanced Ransomware Attack Simulationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026What Makes a Good LLM Agent for Real-world Penetration Testing?
  2. 2026Big Enough to Break Out: Tracking the Rising Capability of LLM Penetration-Testing Agents
  3. 2025Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees
  4. 2025Automated Penetration Testing with LLM Agents and Classical Planning
  5. 2026Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing
  6. 2025AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents
  7. 2026APT-Agent: Automated Penetration Testing using Large Language Models
  8. 2026Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing