# Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs

arXiv: https://arxiv.org/abs/2510.17521 · HTML: https://mlsanity.com/paper/2510.17521/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [PentestAgent: Incorporating LLM Agents to Automated Penetration Testing](https://mlsanity.com/paper/2411.05185/) (2024)
- [HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing](https://mlsanity.com/paper/2412.01778/) (2024)
- [VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2501.13411/) (2025)
- [PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation](https://mlsanity.com/paper/2407.17788/) (2024)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [Hacking CTFs with Plain Agents](https://mlsanity.com/paper/2412.02776/) (2024)
- [CAI: An Open, Bug Bounty-Ready Cybersecurity AI](https://mlsanity.com/paper/2504.06017/) (2025)
- [Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design](https://mlsanity.com/paper/2504.10112/) (2025)
- [Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements](https://mlsanity.com/paper/2410.17141/) (2024)
- [RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents](https://mlsanity.com/paper/2502.16730/) (2025)
- [AutoPT: How Far Are We from the End2End Automated Web Penetration Testing?](https://mlsanity.com/paper/2411.01236/) (2024)
- [A Unified Framework for Human AI Collaboration in Security Operations Centers with Trusted Autonomy](https://mlsanity.com/paper/2505.23397/) (2025)
- [PenTest++: Elevating Ethical Hacking with AI and Automation](https://mlsanity.com/paper/2502.09484/) (2025)
- [Evaluating AI cyber capabilities with crowdsourced elicitation](https://mlsanity.com/paper/2505.19915/) (2025)
- [Cybersecurity AI: The Dangerous Gap Between Automation and Autonomy](https://mlsanity.com/paper/2506.23592/) (2025)
- [CAI Fluency: A Framework for Cybersecurity AI Fluency](https://mlsanity.com/paper/2508.13588/) (2025)
- [Offensive Robot Cybersecurity](https://mlsanity.com/paper/2506.15343/) (2025)
- [Cybersecurity AI: Hacking the AI Hackers via Prompt Injection](https://mlsanity.com/paper/2508.21669/) (2025)

## Built on by (most cited first, 1 of 1)

- [Challenges and Future Directions in Agentic Reverse Engineering Systems](https://mlsanity.com/paper/2604.14317/) (2026)

## Abstract (arXiv)

We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation.