mlsanityData to 7 Oct 2026

Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities

arXiv 2410.09114 ↗First posted 10 Oct 2024cs.CR cs.AI cs.LGcited by 12

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

LLM agents have the potential to revolutionize defensive cyber operations, but their offensive capabilities are not yet fully understood. To prepare for emerging threats, model developers and governments are evaluating the cyber capabilities of foundation models. However, these assessments often lack transparency and a comprehensive focus on offensive capabilities. In response, we introduce the Catastrophic Cyber Capabilities Benchmark (3CB), a novel framework designed to rigorously assess the real-world offensive capabilities of LLM agents. Our evaluation of modern LLMs on 3CB reveals that frontier models, such as GPT-4o and Claude 3.5 Sonnet, can perform offensive tasks such as reconnaissance and exploitation across domains ranging from binary analysis to web technologies. Conversely, smaller open-source models exhibit limited offensive capabilities. Our software solution and the corresponding benchmark provides a critical tool to reduce the gap between rapidly improving capabilities and robustness of cyber offense evaluations, aiding in the safer deployment and regulation of these powerful technologies.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 21 of its 43 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

21 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LLM Agents can Autonomously Hack Websites6 Feb

Other references · 20

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  7. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  8. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  9. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  10. 2023An Overview of Catastrophic AI Riskscited by 214
  11. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  12. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
8 more
  1. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  2. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  3. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  4. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  5. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  6. 2024Escalation Risks from Language Models in Military and Diplomatic Decision-Makingcited by 61
  7. 2024Thousands of AI Authors on the Future of AIcited by 48
  8. 2023Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Frameworkcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 5.

12 papers.

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  2. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 25
  3. 2025Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networkscited by 17
  4. 2024Hacking CTFs with Plain Agentscited by 14
  5. 2025OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilitiescited by 12
  6. 2025Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agentcited by 4
  7. 2026CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environmentscited by 1
  8. 2026CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledgecited by 1
  9. 2026CrackMeBench: Binary Reverse Engineering for Agentscited by 1
  10. 2025From Capabilities to Performance: Evaluating Key Functional Properties of LLM Architectures in Penetration Testingcited by 0
  11. 2026Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Responsecited by 0
  12. 2026Reveree: Diagnosing LLM Reverse-Engineering Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  2. 2026ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents
  3. 2025Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
  4. 2026Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents
  5. 2026LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity
  6. 2025CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
  7. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense
  8. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps