# AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges

arXiv: https://arxiv.org/abs/2606.14295 · HTML: https://mlsanity.com/paper/2606.14295/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems](https://mlsanity.com/paper/2505.15216/) (2025)
- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://mlsanity.com/paper/2605.11086/) (2026)
- [ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents](https://mlsanity.com/paper/2605.14153/) (2026)
- [Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks](https://mlsanity.com/paper/2501.16466/) (2025)
- [Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios](https://mlsanity.com/paper/2603.11214/) (2026)
- [Synthesizing Multi-Agent Harnesses for Vulnerability Discovery](https://mlsanity.com/paper/2604.20801/) (2026)
- [What Makes a Good LLM Agent for Real-world Penetration Testing?](https://mlsanity.com/paper/2602.17622/) (2026)

## Built on by (most cited first, 4 of 4)

- [RangeFactory: Scalable Construction of Multi-Hop Cyber Ranges](https://mlsanity.com/paper/2608.09526/) (2026)
- [Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents](https://mlsanity.com/paper/2608.20563/) (2026)
- [SysEvolve: An AI-native, safe, autonomous adversarial attack-defense co-evolutionary system](https://mlsanity.com/paper/2608.15012/) (2026)
- [SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction](https://mlsanity.com/paper/2609.06229/) (2026)

## Abstract (arXiv)

Frontier AI systems are increasingly capable of cybersecurity tasks, including codebase inspection, vulnerability detection, and exploitation. However, evaluating their offensive capabilities remains constrained by limited access to open, reproducible, multi-host cyber ranges. Existing public benchmarks capture isolated skills such as CTF solving, vulnerability reproduction, and exploit generation, but often abstract away realistic intrusion workflows: discovering exposed services, gaining a foothold, collecting internal information, and expanding compromise across hosts. This gap makes it difficult to observe emerging risks early, because frontier AI systems are rarely evaluated under realistic attack conditions. We introduce AgentCyberRange, the first open, multi-range infrastructure for measuring autonomous cyber attack capability in realistic cyber ranges. It combines 110 vulnerabilities across 15 real web applications and 8 enterprise-like cyber ranges with 156 internal hosts, plus Cage, a toolchain for execution, orchestration, result collection, and verification. The benchmark covers two core stages: web exploitation, where agents explore exposed applications and validate vulnerabilities, and post exploitation, where agents turn an initial foothold into broader internal compromise. We evaluate six frontier AI systems under matched prompts and budgets. GPT-5.5 with Codex performs best, solving 16.1% of web exploitation tasks and 31.7% of post-exploitation tasks; with more concrete hints, these rates increase to 33.0% and 46.3%. We also observe out-of-benchmark findings, including unknown vulnerabilities in popular projects, and payload mutation that bypasses host defenses. These results show that open cyber-range evaluation is necessary for observing emerging offensive capabilities under realistic and reproducible conditions.