# ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents arXiv: https://arxiv.org/abs/2605.14153 ยท HTML: https://mlsanity.com/paper/2605.14153/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 8 of its 27 references that are in the map. - First posted on arXiv 13 May 2026. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 8 of 8) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025) - [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025) - [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024) - [BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems](https://mlsanity.com/paper/2505.15216/) (2025) - [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://mlsanity.com/paper/2605.11086/) (2026) - [Patch-to-PoC: A Systematic Study of Agentic LLM Systems for Linux Kernel N-Day Reproduction](https://mlsanity.com/paper/2602.07287/) (2026) - [LLM-Based Agents for Identifying Bug-Introducing Commits](https://mlsanity.com/paper/2603.29378/) (2026) ## Built on by (most cited first, 19 of 19) - [The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement](https://mlsanity.com/paper/2609.11873/) (2026) - [SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents](https://mlsanity.com/paper/2609.08149/) (2026) - [AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges](https://mlsanity.com/paper/2606.14295/) (2026) - [The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark](https://mlsanity.com/paper/2608.11469/) (2026) - [Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents](https://mlsanity.com/paper/2608.20563/) (2026) - [MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement](https://mlsanity.com/paper/2610.11959/) (2026) - [Transferable Self-Evolving Playbooks for Agentic Security Auditing](https://mlsanity.com/paper/2606.16420/) (2026) - [Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers](https://mlsanity.com/paper/2606.21037/) (2026) - [Chai: Agentic Discovery of Cryptographic Misuse Vulnerabilities](https://mlsanity.com/paper/2606.26933/) (2026) - [StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents](https://mlsanity.com/paper/2607.26314/) (2026) - [SysEvolve: An AI-native, safe, autonomous adversarial attack-defense co-evolutionary system](https://mlsanity.com/paper/2608.15012/) (2026) - [PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?](https://mlsanity.com/paper/2608.26882/) (2026) - [Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Models](https://mlsanity.com/paper/2609.08418/) (2026) - [MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes](https://mlsanity.com/paper/2609.23980/) (2026) - [Evaluating Coding Agents on Kernel Exploit Generation](https://mlsanity.com/paper/2609.25591/) (2026) - [AuraForge: Scaling Security Supervision for Training Coding Agents](https://mlsanity.com/paper/2610.00850/) (2026) - [Natural Code Transformation Confuses Agents in Vulnerability Analysis](https://mlsanity.com/paper/2610.14350/) (2026) - [CWE-bench: Refocusing Agentic Security from Exploitation to Defense](https://mlsanity.com/paper/2610.17020/) (2026) - [ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?](https://mlsanity.com/paper/2610.17837/) (2026) ## Abstract (arXiv) Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target. However, existing LLM security benchmarks treat a crash as exploitation success. That single binary outcome collapses the hard parts of exploitation: the transition from triggering a bug to constructing reusable primitives and control. We present ExploitBench, a capability-graded benchmark that decomposes exploitation into 16 measurable flags, from coverage and crash through sandbox primitives, arbitrary read/write, control-flow hijack, and arbitrary code execution. Each capability is verified by a deterministic oracle that uses a per-run randomized challenge-response for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution. We instantiate ExploitBench on 41 V8 bugs because V8 is both widely deployed and exploitation-hardened. We report three arms: as the primary measurement of model-environment capability, as a secondary arm that adds adaptive coaching to test whether targeted feedback shifts outcomes, and as an ablation that swaps in the model's native CLI to check whether vendor-side optimizations increase exploitation capabilities. Our results show a sharp capability split between publicly deployed frontier models and the private frontier. Across the 8 publicly deployed models tested, reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not. The private model shows arbitrary code execution on approximately half. Overall, results suggest that exploit construction against hardened targets is an emerging frontier capability.