mlsanityData to 7 Oct 2026

Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents

arXiv 2510.24317 ↗First posted 28 Oct 2025cs.CRcited by 11

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Cybersecurity spans multiple interconnected domains, complicating the development of meaningful, labor-relevant benchmarks. Existing benchmarks assess isolated skills rather than integrated performance. We find that pre-trained knowledge of cybersecurity in LLMs does not imply attack and defense abilities, revealing a gap between knowledge and capability. To address this limitation, we present the Cybersecurity AI Benchmark (CAIBench), a modular meta-benchmark framework that allows evaluating LLM models and agents across offensive and defensive cybersecurity domains, taking a step towards meaningfully measuring their labor-relevance. CAIBench integrates five evaluation categories, covering over 10,000 instances: Jeopardy-style CTFs, Attack and Defense CTFs, Cyber Range exercises, knowledge benchmarks, and privacy assessments. Key novel contributions include systematic simultaneous offensive-defensive evaluation, robotics-focused cybersecurity challenges (RCTF2), and privacy-preserving performance assessment (CyberPII-Bench). Evaluation of state-of-the-art AI models reveals saturation on security knowledge metrics (~70% success) but substantial degradation in multi-step adversarial (A&D) scenarios (20-40% success), or worse in robotic targets (22% success). The combination of framework scaffolding and LLM model choice significantly impacts performance; we find that proper matches improve up to 2.6× variance in Attack and Defense CTFs. These results demonstrate a pronounced gap between conceptual knowledge and adaptive capability, emphasizing the need for a meta-benchmark.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 12 of its 20 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

12 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024AutoPenBench: Benchmarking Generative Agents for Penetration Testing4 Oct

Other references · 11

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 94
  5. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  6. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  7. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  8. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  9. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
  10. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
  11. 2018Robotics CTF (RCTF), a playground for robot hackingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefensecited by 3
  2. 2026SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agentscited by 3
  3. 2026CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilitiescited by 2
  4. 2026KryptoPilot: An Open-World Knowledge-Augmented LLM Agent for Automated Cryptographic Exploitationcited by 1
  5. 2026CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environmentscited by 1
  6. 2026Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agentscited by 1
  7. 2026Can AI Lower the Barrier to Cybersecurity? A Human-Centered Mixed-Methods Study of Novice CTF Learningcited by 0
  8. 2026From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-Worldcited by 0
  9. 2026Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarkscited by 0
  10. 2026Evaluating Context Segmentation in Locally Deployable SLMs for Cybersecurity CTF Taskscited by 0
  11. 2026CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs
  2. 2025CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
  3. 2026A New Framework for Cybersecurity Refusals in AI Agents
  4. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding
  5. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
  6. 2025The Evolution of Agentic AI in Cybersecurity: From Single LLM Reasoners to Multi-Agent Systems and Autonomous Pipelines
  7. 2025SoK: Measuring What Matters for Closed-Loop Security Agents
  8. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence