mlsanityData to 14 Oct 2026

ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense

arXiv 2603.02297 ↗First posted 2 Mar 2026cs.CR cs.AIcited by 4

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models (LLMs) are increasingly being deployed as software engineering agents that autonomously contribute to repositories. A major benefit these agents present is their ability to find and patch security vulnerabilities in the codebases they oversee. To estimate the capability of agents in this domain, we introduce ZeroDayBench, a benchmark where LLM agents find and patch 22 novel critical vulnerabilities in open-source codebases. We focus our efforts on three popular frontier agentic LLMs: GPT-5.2, Claude Sonnet 4.5, and Grok 4.1. We find that frontier LLMs are not yet capable of autonomously solving our tasks and observe some behavioral patterns that suggest how these models can be improved in the domain of proactive cyberdefense.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 15 of its 25 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

15 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents28 Oct
  2. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
  3. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities21 Mar
  4. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories5 Mar

Other references · 11

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  4. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 78
  5. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  6. 2025PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilitiescited by 21
  7. 2025SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detectioncited by 19
  8. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 14
  9. 2025VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilitiescited by 12
  10. 2025VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediationcited by 5
  11. 2026StriderSPD: Structure-Guided Joint Representation Learning for Binary Security Patch Detectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 9
  2. 2026Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarkscited by 0
  3. 2026FORGE: Multi-Agent Graduated Exploitation and Detection Engineeringcited by 0
  4. 2026CWE-bench: Refocusing Agentic Security from Exploitation to Defensecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?
  2. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  3. 2025Cyber-Zero: Training Cybersecurity Agents without Runtime
  4. 2025From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection
  5. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  6. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities
  7. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  8. 2024LLM Agents can Autonomously Hack Websites