# PentestAgent: Incorporating LLM Agents to Automated Penetration Testing

arXiv: https://arxiv.org/abs/2411.05185 · HTML: https://mlsanity.com/paper/2411.05185/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 42 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 8 of 8)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- ["We Need Structured Output": Towards User-centered Constraints on Large Language Model Output](https://mlsanity.com/paper/2404.07362/) (2024)
- [POMDPs Make Better Hackers: Accounting for Uncertainty in Penetration Testing](https://mlsanity.com/paper/1307.8182/) (2013)
- [Penetration Testing == POMDP Solving?](https://mlsanity.com/paper/1306.4714/) (2013)
- [Attack Planning in the Real World](https://mlsanity.com/paper/1306.4044/) (2013)
- [An Algorithm to Find Optimal Attack Paths in Nondeterministic Scenarios](https://mlsanity.com/paper/1306.4040/) (2013)

## Built on by (most cited first, 30 of 47)

- [D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security](https://mlsanity.com/paper/2502.10931/) (2025)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System](https://mlsanity.com/paper/2505.04254/) (2025)
- [CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution](https://mlsanity.com/paper/2505.17107/) (2025)
- [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025)
- [What Makes a Good LLM Agent for Real-world Penetration Testing?](https://mlsanity.com/paper/2602.17622/) (2026)
- [AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents](https://mlsanity.com/paper/2505.10321/) (2025)
- [RedTeamLLM: an Agentic AI framework for offensive security](https://mlsanity.com/paper/2505.06913/) (2025)
- [PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities](https://mlsanity.com/paper/2510.11688/) (2025)
- [PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design](https://mlsanity.com/paper/2512.14233/) (2025)
- [Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting](https://mlsanity.com/paper/2509.23571/) (2025)
- [AutoPentester: An LLM Agent-based Framework for Automated Pentesting](https://mlsanity.com/paper/2510.05605/) (2025)
- [Automated Penetration Testing with LLM Agents and Classical Planning](https://mlsanity.com/paper/2512.11143/) (2025)
- [Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation](https://mlsanity.com/paper/2508.07745/) (2025)
- [From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization](https://mlsanity.com/paper/2510.02389/) (2025)
- [LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?](https://mlsanity.com/paper/2510.14700/) (2025)
- [Hiding in the AI Traffic: Abusing MCP for LLM-Powered Agentic Red Teaming](https://mlsanity.com/paper/2511.15998/) (2025)
- [CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics](https://mlsanity.com/paper/2508.20643/) (2025)
- [xOffense: An Autonomous Multi-Agent Framework for Penetration Testing with Domain-Adapted Large Language Models](https://mlsanity.com/paper/2509.13021/) (2025)
- [RvB: Automating AI System Hardening via Iterative Red-Blue Games](https://mlsanity.com/paper/2601.19726/) (2026)
- [CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking](https://mlsanity.com/paper/2602.08023/) (2026)
- [Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06921/) (2025)
- [APT-Agent: Automated Penetration Testing using Large Language Models](https://mlsanity.com/paper/2605.24949/) (2026)
- [Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs](https://mlsanity.com/paper/2510.17521/) (2025)
- [PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing](https://mlsanity.com/paper/2601.06910/) (2026)
- [Red-MIRROR: Agentic LLM-based Autonomous Penetration Testing with Reflective Verification and Knowledge-augmented Interaction](https://mlsanity.com/paper/2603.27127/) (2026)
- [Challenges and Future Directions in Agentic Reverse Engineering Systems](https://mlsanity.com/paper/2604.14317/) (2026)
- [Automation-Exploit: A Multi-Agent LLM Framework for Adaptive Offensive Security with Digital Twin-Based Risk-Mitigated Exploitation](https://mlsanity.com/paper/2604.22427/) (2026)
- [Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots](https://mlsanity.com/paper/2605.29963/) (2026)
- [How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency](https://mlsanity.com/paper/2605.30096/) (2026)

## Abstract (arXiv)

Penetration testing is a critical technique for identifying security vulnerabilities, traditionally performed manually by skilled security specialists. This complex process involves gathering information about the target system, identifying entry points, exploiting the system, and reporting findings. Despite its effectiveness, manual penetration testing is time-consuming and expensive, often requiring significant expertise and resources that many organizations cannot afford. While automated penetration testing methods have been proposed, they often fall short in real-world applications due to limitations in flexibility, adaptability, and implementation. Recent advancements in large language models (LLMs) offer new opportunities for enhancing penetration testing through increased intelligence and automation. However, current LLM-based approaches still face significant challenges, including limited penetration testing knowledge and a lack of comprehensive automation capabilities. To address these gaps, we propose PentestAgent, a novel LLM-based automated penetration testing framework that leverages the power of LLMs and various LLM-based techniques like Retrieval Augmented Generation (RAG) to enhance penetration testing knowledge and automate various tasks. Our framework leverages multi-agent collaboration to automate intelligence gathering, vulnerability analysis, and exploitation stages, reducing manual intervention. We evaluate PentestAgent using a comprehensive benchmark, demonstrating superior performance in task completion and overall efficiency. This work significantly advances the practical applicability of automated penetration testing systems.