mlsanityData to 7 Oct 2026

CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

arXiv 2506.02548 ↗First posted 3 Jun 2025cs.CR cs.AI cs.LGcited by 71

Not on the map. Why?

Cited by 42 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

AI agents have significant potential to reshape cybersecurity, making a thorough assessment of their capabilities critical. However, existing evaluations fall short, because they are based on small-scale benchmarks and only measure static outcomes, failing to capture the full, dynamic range of real-world security challenges. To address these limitations, we introduce CyberGym, a large-scale benchmark featuring 1,507 real-world vulnerabilities across 188 software projects. Adjustable to different vulnerability analysis settings, CyberGym primarily tasks agents with generating a proof-of-concept test that reproduces a vulnerability, given only its text description and the corresponding codebase. Our extensive evaluation highlights that CyberGym effectively differentiates agents' and models' cybersecurity capabilities. Even the top-performing combinations only achieve a ~20% success rate, demonstrating the overall difficulty of CyberGym. Beyond static benchmarking, we show that CyberGym leads to the discovery of 34 zero-day vulnerabilities and 18 historically incomplete patches. These results underscore that CyberGym is not only a robust benchmark for measuring AI's progress in cybersecurity but also a platform for creating direct, real-world security impact.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 16 of its 56 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

16 of 56 references.

References · 16

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  4. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  5. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  6. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  7. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  8. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  9. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  10. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  11. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 37
  12. 2016A Survey of Symbolic Execution Techniquescited by 36
4 more
  1. 2018Evaluating Fuzz Testingcited by 29
  2. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  3. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 14
  4. 2023What Happens When We Fuzz? Investigating OSS-Fuzz Bug Historycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 65.

71 papers.

Sort:

By area:

  1. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 24
  2. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 19
  3. 2026Quantifying Frontier LLM Capabilities for Container Sandbox Escapecited by 18
  4. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  5. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 8
  6. 2026CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilitiescited by 6
  7. 2026The Last AI Built by Humans: Toward Genuine Recursive Self-Improvementcited by 5
  8. 2025Agentic Discovery and Validation of Android App Vulnerabilitiescited by 4
  9. 2025Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agentcited by 4
  10. 2026Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?cited by 4
  11. 2026AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Rangescited by 4
  12. 2026OmniCode: A Benchmark for Evaluating Software Engineering Agentscited by 3
38 more
  1. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefensecited by 3
  2. 2026NVIDIA-labs OO Agents: Native Python Object-Oriented Agentscited by 3
  3. 2026Quantifying Self-Preservation Bias in Large Language Modelscited by 2
  4. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 2
  5. 2026Antares: Foundation Models for Agentic Vulnerability Localizationcited by 2
  6. 2025CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptographycited by 1
  7. 2026Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalationcited by 1
  8. 2026CrackMeBench: Binary Reverse Engineering for Agentscited by 1
  9. 2026No More, No Less: Task Alignment in Terminal Agentscited by 1
  10. 2026Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypotscited by 1
  11. 2026A New Framework for Cybersecurity Refusals in AI Agentscited by 1
  12. 2026SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agentscited by 1
  13. 2026RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilitiescited by 1
  14. 2026RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discoverycited by 1
  15. 2026Agentic Incident Response through Digital Twin-Enhanced Multiscale Planningcited by 1
  16. 2026AI Security Leaderboard: Methodology, Results and Minimal Standardcited by 1
  17. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  18. 2026RangeFactory: Scalable Construction of Multi-Hop Cyber Rangescited by 1
  19. 2026The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmarkcited by 1
  20. 2025Uplifted Attackers, Human Defenders: The Cyber Offense-Defense Balance for Trailing-Edge Organizationscited by 0
  21. 2026Scalable Delphi: Large Language Models for Structured Risk Estimationcited by 0
  22. 2026Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systemscited by 0
  23. 2026Agentic Vulnerability Reasoning on COTS Binariescited by 0
  24. 2026Agentic AI and the Industrialization of Cyber Offense: Forecast, Consequences, and Defensive Priorities for Enterprises and the Mittelstandcited by 0
  25. 2026From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-Worldcited by 0
  26. 2026FragBench: Cross-Session Attacks Hidden in Benign-Looking Fragmentscited by 0
  27. 2026GenAI-Driven Threat Detection with Microsoft Security Copilotcited by 0
  28. 2026Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hardcited by 0
  29. 2026CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Flycited by 0
  30. 2026Transferable Self-Evolving Playbooks for Agentic Security Auditingcited by 0
  31. 2026CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?cited by 0
  32. 2026Chai: Agentic Discovery of Cryptographic Misuse Vulnerabilitiescited by 0
  33. 2026Two AI Metrics Diverged: Will it Make All the Difference?cited by 0
  34. 2026Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproductioncited by 0
  35. 2026Harmonizing AI Safety Thresholdscited by 0
  36. 2026CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Trainingcited by 0
  37. 2026Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesiscited by 0
  38. 2026SysEvolve: An AI-native, safe, autonomous adversarial attack-defense co-evolutionary systemcited by 0

50 most cited of 71 All 71 (JSON)