# From Capabilities to Performance: Evaluating Key Functional Properties of LLM Architectures in Penetration Testing

arXiv: https://arxiv.org/abs/2509.14289 · HTML: https://mlsanity.com/paper/2509.14289/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 70 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)](https://mlsanity.com/paper/2505.02279/) (2025)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing](https://mlsanity.com/paper/2412.01778/) (2024)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directions](https://mlsanity.com/paper/2405.14487/) (2024)
- [INMS: Memory Sharing for Large Language Model based Agents](https://mlsanity.com/paper/2404.09982/) (2024)
- [PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation](https://mlsanity.com/paper/2407.17788/) (2024)
- [Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions](https://mlsanity.com/paper/2308.10443/) (2023)
- [Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities](https://mlsanity.com/paper/2405.12750/) (2024)
- [RatGPT: Turning online LLMs into Proxies for Malware Attacks](https://mlsanity.com/paper/2308.09183/) (2023)
- [Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments](https://mlsanity.com/paper/2308.12086/) (2023)
- [LLMCloudHunter: Harnessing LLMs for Automated Extraction of Detection Rules from Cloud-Based CTI](https://mlsanity.com/paper/2407.05194/) (2024)
- [LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing](https://mlsanity.com/paper/2310.06936/) (2023)
- [Multi-Agent Coordination via Multi-Level Communication](https://mlsanity.com/paper/2209.12713/) (2022)
- [From Chatbots to PhishBots? -- Preventing Phishing scams created using ChatGPT, Google Bard and Claude](https://mlsanity.com/paper/2310.19181/) (2023)
- [Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities](https://mlsanity.com/paper/2410.09114/) (2024)
- [Discovering Exfiltration Paths Using Reinforcement Learning with Attack Graphs](https://mlsanity.com/paper/2201.12416/) (2022)
- [A Survey of Large Language Models in Cybersecurity](https://mlsanity.com/paper/2402.16968/) (2024)
- [Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications](https://mlsanity.com/paper/2406.11007/) (2024)
- [Position: AI Safety Must Embrace an Antifragile Perspective](https://mlsanity.com/paper/2509.13339/) (2025)
- [Exposing Surveillance Detection Routes via Reinforcement Learning, Attack Graphs, and Cyber Terrain](https://mlsanity.com/paper/2211.03027/) (2022)
- [Fast and Adaptive Multi-agent Planning under Collaborative Temporal Logic Tasks via Poset Products](https://mlsanity.com/paper/2308.11373/) (2023)
- [Leveraging Reinforcement Learning in Red Teaming for Advanced Ransomware Attack Simulations](https://mlsanity.com/paper/2406.17576/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to automate or augment penetration testing, but their effectiveness and reliability across attack phases remain unclear. We present a comprehensive evaluation of multiple LLM-based agents, from single-agent to modular designs, across realistic penetration testing scenarios, measuring empirical performance and recurring failure patterns. We also isolate the impact of five core functional capabilities via targeted augmentations: Global Context Memory (GCM), Inter-Agent Messaging (IAM), Context-Conditioned Invocation (CCI), Adaptive Planning (AP), and Real-Time Monitoring (RTM). These interventions support, respectively: (i) context coherence and retention, (ii) inter-component coordination and state management, (iii) tool use accuracy and selective execution, (iv) multi-step strategic planning, error detection, and recovery, and (v) real-time dynamic responsiveness. Our results show that while some architectures natively exhibit subsets of these properties, targeted augmentations substantially improve modular agent performance, especially in complex, multi-step, and real-time penetration testing tasks.