# On the Surprising Efficacy of LLMs for Penetration-Testing

arXiv: https://arxiv.org/abs/2507.00829 · HTML: https://mlsanity.com/paper/2507.00829/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2025.

## Built on by (most cited first, 10 of 10)

- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [Hiding in the AI Traffic: Abusing MCP for LLM-Powered Agentic Red Teaming](https://mlsanity.com/paper/2511.15998/) (2025)
- [Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing](https://mlsanity.com/paper/2604.05719/) (2026)
- [STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving](https://mlsanity.com/paper/2603.22577/) (2026)
- [Red-MIRROR: Agentic LLM-based Autonomous Penetration Testing with Reflective Verification and Knowledge-augmented Interaction](https://mlsanity.com/paper/2603.27127/) (2026)
- [CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environments](https://mlsanity.com/paper/2604.06019/) (2026)
- [Servant, Stalker, Predator: How An Honest, Helpful, And Harmless (3H) Agent Unlocks Adversarial Skills](https://mlsanity.com/paper/2508.19500/) (2025)
- [Can AI Lower the Barrier to Cybersecurity? A Human-Centered Mixed-Methods Study of Novice CTF Learning](https://mlsanity.com/paper/2602.18172/) (2026)
- [Automating Attack Graph Construction for Agentic Pentesting. Towards Neuro-Symbolic Vulnerability Hunting](https://mlsanity.com/paper/2609.15523/) (2026)
- [PentestChain: A Cost-Aware, MCP-Orchestrated Framework for Automated Penetration Testing with Free-Tier LLMs](https://mlsanity.com/paper/2609.18120/) (2026)

## Abstract (arXiv)

This paper presents a critical examination of the surprising efficacy of Large Language Models (LLMs) in penetration testing. The paper thoroughly reviews the evolution of LLMs and their rapidly expanding capabilities which render them increasingly suitable for complex penetration testing operations. It systematically details the historical adoption of LLMs in both academic research and industry, showcasing their application across various offensive security tasks and covering broader phases of the cyber kill chain. Crucially, the analysis also extends to the observed adoption of LLMs by malicious actors, underscoring the inherent dual-use challenge of this technology within the security landscape. The unexpected effectiveness of LLMs in this context is elucidated by several key factors: the strong alignment between penetration testing's reliance on pattern-matching and LLMs' core strengths, their inherent capacity to manage uncertainty in dynamic environments, and cost-effective access to competent pre-trained models through LLM providers. The current landscape of LLM-aided penetration testing is categorized into interactive 'vibe-hacking' and the emergence of fully autonomous systems. The paper identifies and discusses significant obstacles impeding wider adoption and safe deployment. These include critical issues concerning model reliability and stability, paramount safety and security concerns, substantial monetary and ecological costs, implications for privacy and digital sovereignty, complex questions of accountability, and profound ethical dilemmas. This comprehensive review and analysis provides a foundation for discussion on future research directions and the development of robust safeguards at the intersection of AI and security.