# PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design

arXiv: https://arxiv.org/abs/2512.14233 · HTML: https://mlsanity.com/paper/2512.14233/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 46 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 9 of 9)

- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [PentestAgent: Incorporating LLM Agents to Automated Penetration Testing](https://mlsanity.com/paper/2411.05185/) (2024)
- [A Survey on Large Language Models with some Insights on their Capabilities and Limitations](https://mlsanity.com/paper/2501.04040/) (2025)
- [Autonomous Penetration Testing using Reinforcement Learning](https://mlsanity.com/paper/1905.05965/) (2019)
- [VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2501.13411/) (2025)
- [PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation](https://mlsanity.com/paper/2407.17788/) (2024)
- [AutoPenBench: Benchmarking Generative Agents for Penetration Testing](https://mlsanity.com/paper/2410.03225/) (2024)
- [Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements](https://mlsanity.com/paper/2410.17141/) (2024)

## Built on by (most cited first, 7 of 7)

- [Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing](https://mlsanity.com/paper/2604.05719/) (2026)
- [Automation-Exploit: A Multi-Agent LLM Framework for Adaptive Offensive Security with Digital Twin-Based Risk-Mitigated Exploitation](https://mlsanity.com/paper/2604.22427/) (2026)
- [Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents](https://mlsanity.com/paper/2608.20563/) (2026)
- [From Capability to Assurance in Autonomous Penetration-Testing Harnesses: A Framework and Reference Implementation](https://mlsanity.com/paper/2609.22664/) (2026)
- [AutoEG: Exploiting Known Third-Party Vulnerabilities in Black-Box Web Applications](https://mlsanity.com/paper/2604.00704/) (2026)
- [From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World](https://mlsanity.com/paper/2605.10834/) (2026)
- [Staying on the Attack Path: Structured State for Long-Horizon Automated Penetration Testing](https://mlsanity.com/paper/2609.07344/) (2026)

## Abstract (arXiv)

Penetration testing is essential for assessing and strengthening system security against real-world threats, yet traditional workflows remain highly manual, expertise-intensive, and difficult to scale. Although recent advances in Large Language Models (LLMs) offer promising opportunities for automation, existing applications rely on simplistic prompting without task decomposition or domain adaptation, resulting in unreliable black-box behavior and limited insight into model capabilities across penetration testing stages. To address this gap, we introduce PentestEval, the first comprehensive benchmark for evaluating LLMs across six decomposed penetration testing stages: Information Collection, Weakness Gathering and Filtering, Attack Decision-Making, Exploit Generation and Revision. PentestEval integrates expert-annotated ground truth with a fully automated evaluation pipeline across 346 tasks covering all stages in 12 realistic vulnerable scenarios. Our stage-level evaluation of 9 widely used LLMs reveals generally weak performance and distinct limitations across the stages of penetration-testing workflow. End-to-end pipelines reach only 31% success rate, and existing LLM-powered systems such as PentestGPT, PentestAgent, and VulnBot exhibit similar limitations, with autonomous agents failing almost entirely. These findings highlight that autonomous penetration testing demands stronger structured reasoning, where modularization enhances each individual stage and improves overall performance. PentestEval provides the foundational benchmark needed for future research on fine-grained, stage-level evaluation, paving the way toward more reliable LLM-based automation.