mlsanityData to 7 Oct 2026

CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities

arXiv 2503.17332 ↗First posted 21 Mar 2025cs.CR cs.AIcited by 90

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks, posing significant threats to existing applications. This growing risk highlights the urgent need for a real-world benchmark to evaluate the ability of LLM agents to exploit web application vulnerabilities. However, existing benchmarks fall short as they are limited to abstracted Capture the Flag competitions or lack comprehensive coverage. Building a benchmark for real-world vulnerabilities involves both specialized expertise to reproduce exploits and a systematic approach to evaluating unpredictable threats. To address this challenge, we introduce CVE-Bench, a real-world cybersecurity benchmark based on critical-severity Common Vulnerabilities and Exposures. In CVE-Bench, we design a sandbox framework that enables LLM agents to exploit vulnerable web applications in scenarios that mimic real-world conditions, while also providing effective evaluation of their exploits. Our evaluation shows that the state-of-the-art agent framework can resolve up to 13% of vulnerabilities.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 19 of its 86 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

19 of 86 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  3. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities2 Jun
  4. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilities11 Apr
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  6. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  7. 2024LLM Agents can Autonomously Hack Websites6 Feb

Other references · 12

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  7. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  8. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  9. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  10. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 56
  11. 2024Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Aheadcited by 41
  12. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 59.

90 papers.

Sort:

By area:

  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  2. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  3. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  4. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  5. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  6. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 37
  7. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  8. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 24
  9. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  10. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 18
  11. 2026Quantifying Frontier LLM Capabilities for Container Sandbox Escapecited by 18
  12. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
38 more
  1. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  2. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  3. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  4. 2026ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agentscited by 16
  5. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  6. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  7. 2025Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testingcited by 14
  8. 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectoriescited by 12
  9. 2025FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agentscited by 11
  10. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 10
  11. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 8
  12. 2026EVMbench: Evaluating AI Agents on Smart Contract Securitycited by 7
  13. 2025PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilitiescited by 6
  14. 2026CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilitiescited by 6
  15. 2025LLMs unlock new paths to monetizing exploitscited by 5
  16. 2025VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediationcited by 5
  17. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  18. 2026CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerabilitycited by 5
  19. 2026Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Constructioncited by 5
  20. 2025Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agentcited by 4
  21. 2025AutoBaxBuilder: Bootstrapping Code Security Benchmarkingcited by 4
  22. 2025With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systemscited by 4
  23. 2026AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Rangescited by 4
  24. 2025CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensicscited by 3
  25. 2025SecureFixAgent: A Hybrid LLM Agent for Automated Python Static Vulnerability Repaircited by 3
  26. 2026OmniCode: A Benchmark for Evaluating Software Engineering Agentscited by 3
  27. 2026Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generationcited by 3
  28. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefensecited by 3
  29. 2026Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challengescited by 3
  30. 2025Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Securitycited by 2
  31. 2025Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunitiescited by 2
  32. 2025xOffense: An Autonomous Multi-Agent Framework for Penetration Testing with Domain-Adapted Large Language Modelscited by 2
  33. 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 2
  34. 2026VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?cited by 2
  35. 2026Autonomous LLM Agents & CTFs: A Second Lookcited by 2
  36. 2026AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agentscited by 2
  37. 2026PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testingcited by 1
  38. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?
  2. 2025SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents
  3. 2026AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions
  4. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
  5. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  6. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
  7. 2025HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities
  8. 2025CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning