# AutoPenBench: Benchmarking Generative Agents for Penetration Testing arXiv: https://arxiv.org/abs/2410.03225 ยท HTML: https://mlsanity.com/paper/2410.03225/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 10 of its 31 references that are in the map. - First posted on arXiv 4 Oct 2024. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 10 of 10) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023) - [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024) - [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024) - [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023) - [Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs](https://mlsanity.com/paper/2305.14279/) (2023) - [Towards interactive evaluations for interaction harms in human-AI systems](https://mlsanity.com/paper/2405.10632/) (2024) - [AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents](https://mlsanity.com/paper/2404.06411/) (2024) - [Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering](https://mlsanity.com/paper/2310.17490/) (2023) ## Built on by (most cited first, 21 of 21) - [VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2501.13411/) (2025) - [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025) - [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025) - [Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design](https://mlsanity.com/paper/2504.10112/) (2025) - [Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents](https://mlsanity.com/paper/2510.24317/) (2025) - [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025) - [What Makes a Good LLM Agent for Real-world Penetration Testing?](https://mlsanity.com/paper/2602.17622/) (2026) - [PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities](https://mlsanity.com/paper/2510.11688/) (2025) - [PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design](https://mlsanity.com/paper/2512.14233/) (2025) - [Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning](https://mlsanity.com/paper/2508.07382/) (2025) - [SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI](https://mlsanity.com/paper/2410.11096/) (2024) - [Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06921/) (2025) - [The Evolution of Agentic AI in Cybersecurity: From Single LLM Reasoners to Multi-Agent Systems and Autonomous Pipelines](https://mlsanity.com/paper/2512.06659/) (2025) - [To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack](https://mlsanity.com/paper/2602.02595/) (2026) - [CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environments](https://mlsanity.com/paper/2604.06019/) (2026) - [SoK: Measuring What Matters for Closed-Loop Security Agents](https://mlsanity.com/paper/2510.01654/) (2025) - [From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World](https://mlsanity.com/paper/2605.10834/) (2026) - [CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly](https://mlsanity.com/paper/2605.26195/) (2026) - [The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems](https://mlsanity.com/paper/2606.13079/) (2026) - [Reveree: Diagnosing LLM Reverse-Engineering Agents](https://mlsanity.com/paper/2609.01185/) (2026) - [PentestChain: A Cost-Aware, MCP-Orchestrated Framework for Automated Penetration Testing with Free-Tier LLMs](https://mlsanity.com/paper/2609.18120/) (2026) ## Abstract (arXiv) Generative AI agents, software systems powered by Large Language Models (LLMs), are emerging as a promising approach to automate cybersecurity tasks. Among the others, penetration testing is a challenging field due to the task complexity and the diverse strategies to simulate cyber-attacks. Despite growing interest and initial studies in automating penetration testing with generative agents, there remains a significant gap in the form of a comprehensive and standard framework for their evaluation and development. This paper introduces AutoPenBench, an open benchmark for evaluating generative agents in automated penetration testing. We present a comprehensive framework that includes 33 tasks, each representing a vulnerable system that the agent has to attack. Tasks are of increasing difficulty levels, including in-vitro and real-world scenarios. We assess the agent performance with generic and specific milestones that allow us to compare results in a standardised manner and understand the limits of the agent under test. We show the benefits of AutoPenBench by testing two agent architectures: a fully autonomous and a semi-autonomous supporting human interaction. We compare their performance and limitations. For example, the fully autonomous agent performs unsatisfactorily achieving a 21% Success Rate (SR) across the benchmark, solving 27% of the simple tasks and only one real-world task. In contrast, the assisted agent demonstrates substantial improvements, with 64% of SR. AutoPenBench allows us also to observe how different LLMs like GPT-4o or OpenAI o1 impact the ability of the agents to complete the tasks. We believe that our benchmark fills the gap with a standard and flexible framework to compare penetration testing agents on a common ground. We hope to extend AutoPenBench along with the research community by making it available under https://github.com/lucagioacchini/auto-pen-bench.