mlsanityData to 7 Oct 2026

Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

arXiv 2512.09882 ↗First posted 10 Dec 2025cs.AI cs.CR cs.CYcited by 14

Not on the map. Why?

Abstract (arXiv)

We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment. We evaluate ten cybersecurity professionals alongside six existing AI agents and ARTEMIS, our new agent scaffold, on a large university network consisting of ~8,000 hosts across 12 subnets. ARTEMIS is a multi-agent framework featuring dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. In our comparative study, ARTEMIS placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants. While existing scaffolds such as Codex and CyAgent underperformed relative to most human participants, ARTEMIS demonstrated technical sophistication and submission quality comparable to the strongest participants. We observe that AI agents offer advantages in systematic enumeration, parallel exploitation, and cost -- certain ARTEMIS variants cost 18/hour versus 60/hour for professional penetration testers. We also identify key capability gaps: AI agents exhibit higher false-positive rates and struggle with GUI-based tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

18 of 44 references.

References · 18

  1. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  2. 2024LLM Agents can Autonomously Hack Websitescited by 108
  3. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 94
  4. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  5. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  6. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  7. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  8. 2024TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generationcited by 50
  9. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 37
  10. 2023How Far Have We Gone in Vulnerability Detection Using Large Language Modelscited by 27
  11. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  12. 2025Multi-Agent Penetration Testing AI for the Webcited by 19
6 more
  1. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  2. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 15
  3. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 14
  4. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  5. 2025Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testingcited by 7
  6. 2025Evaluating AI cyber capabilities with crowdsourced elicitationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  2. 2026Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarioscited by 13
  3. 2026The End of Code Review: Coding Agents Supersede Human Inspectioncited by 3
  4. 2026Autonomous LLM Agents & CTFs: A Second Lookcited by 2
  5. 2026AI Agents Enable Adaptive Computer Wormscited by 2
  6. 2026A New Framework for Cybersecurity Refusals in AI Agentscited by 1
  7. 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
  8. 2025The Silicon Psyche: Anthropomorphic Vulnerabilities in Large Language Modelscited by 0
  9. 2026Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarkscited by 0
  10. 2026CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Flycited by 0
  11. 2026Stateful Online Monitoring Catches Distributed Agent Attackscited by 0
  12. 2026Red-Teaming the Agentic Red-Teamcited by 0
2 more
  1. 2026Position: AI Lock-In Is in Progress, and We Must Be Preparedcited by 0
  2. 2026CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenancecited by 0