# PentestGPT: An LLM-empowered Automatic Penetration Testing Tool

arXiv: https://arxiv.org/abs/2308.06782 · HTML: https://mlsanity.com/paper/2308.06782/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2023.

## Built on by (most cited first, 30 of 97)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities](https://mlsanity.com/paper/2409.16165/) (2024)
- [HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing](https://mlsanity.com/paper/2412.01778/) (2024)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks](https://mlsanity.com/paper/2507.10535/) (2025)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks](https://mlsanity.com/paper/2310.11409/) (2023)
- [LLM for SoC Security: A Paradigm Shift](https://mlsanity.com/paper/2310.06046/) (2023)
- [LLMs for Relational Reasoning: How Far are We?](https://mlsanity.com/paper/2401.09042/) (2024)
- [Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptions](https://mlsanity.com/paper/2310.02431/) (2023)
- [Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games](https://mlsanity.com/paper/2403.17674/) (2024)
- [A Framework for Evaluating Emerging Cyberattack Capabilities of AI](https://mlsanity.com/paper/2503.11917/) (2025)
- [Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks](https://mlsanity.com/paper/2410.20911/) (2024)
- [AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses](https://mlsanity.com/paper/2503.01811/) (2025)
- [Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing](https://mlsanity.com/paper/2512.09882/) (2025)
- [CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models](https://mlsanity.com/paper/2410.21060/) (2024)
- [CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution](https://mlsanity.com/paper/2505.17107/) (2025)
- [OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities](https://mlsanity.com/paper/2502.15797/) (2025)
- [Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents](https://mlsanity.com/paper/2510.24317/) (2025)
- [Cyber Sentinel: Exploring Conversational Agents in Streamlining Security Tasks with GPT-4](https://mlsanity.com/paper/2309.16422/) (2023)
- [Review of Generative AI Methods in Cybersecurity](https://mlsanity.com/paper/2403.08701/) (2024)
- [Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environments](https://mlsanity.com/paper/2409.11276/) (2024)

## Abstract (arXiv)

Penetration testing, a crucial industrial practice for ensuring system security, has traditionally resisted automation due to the extensive expertise required by human professionals. Large Language Models (LLMs) have shown significant advancements in various domains, and their emergent abilities suggest their potential to revolutionize industries. In this research, we evaluate the performance of LLMs on real-world penetration testing tasks using a robust benchmark created from test machines with platforms. Our findings reveal that while LLMs demonstrate proficiency in specific sub-tasks within the penetration testing process, such as using testing tools, interpreting outputs, and proposing subsequent actions, they also encounter difficulties maintaining an integrated understanding of the overall testing scenario. In response to these insights, we introduce PentestGPT, an LLM-empowered automatic penetration testing tool that leverages the abundant domain knowledge inherent in LLMs. PentestGPT is meticulously designed with three self-interacting modules, each addressing individual sub-tasks of penetration testing, to mitigate the challenges related to context loss. Our evaluation shows that PentestGPT not only outperforms LLMs with a task-completion increase of 228.6% compared to the gptthree model among the benchmark targets but also proves effective in tackling real-world penetration testing challenges. Having been open-sourced on GitHub, PentestGPT has garnered over 4,700 stars and fostered active community engagement, attesting to its value and impact in both the academic and industrial spheres.