# CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

arXiv: https://arxiv.org/abs/2606.04460 · HTML: https://mlsanity.com/paper/2606.04460/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation](https://mlsanity.com/paper/2501.08200/) (2025)
- [BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems](https://mlsanity.com/paper/2505.15216/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories](https://mlsanity.com/paper/2504.21205/) (2025)

## Built on by (most cited first, 8 of 8)

- [SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities](https://mlsanity.com/paper/2609.33763/) (2026)
- [Red-Teaming the Agentic Red-Team](https://mlsanity.com/paper/2606.24496/) (2026)
- [CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild](https://mlsanity.com/paper/2608.23181/) (2026)
- [Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale](https://mlsanity.com/paper/2609.15939/) (2026)
- [MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes](https://mlsanity.com/paper/2609.23980/) (2026)
- [AuraForge: Scaling Security Supervision for Training Coding Agents](https://mlsanity.com/paper/2610.00850/) (2026)
- [CWE-bench: Refocusing Agentic Security from Exploitation to Defense](https://mlsanity.com/paper/2610.17020/) (2026)
- [EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents](https://mlsanity.com/paper/2610.19433/) (2026)

## Abstract (arXiv)

AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity evaluations of AI systems are limited in scale or scope, and fail to capture the end-to-end lifecycle of real-world software vulnerability discovery and remediation. To address this gap, we propose CyberGym-E2E, a large-scale and realistic end-to-end cybersecurity benchmark that comprehensively evaluates AI agents' abilities across the full lifecycle of vulnerability discovery, PoC generation, and patch generation. CyberGym-E2E is comprehensive and scalable, as we build an automated, agent-enhanced pipeline for transforming open-source vulnerability data into realistic evaluation environments. Currently, the benchmark consists of 920 real-world vulnerabilities across 139 different open-source projects.