mlsanityData to 7 Oct 2026

CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models

arXiv 2408.01605 ↗First posted 2 Aug 2024cs.CR cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and capabilities. CYBERSECEVAL 3 assesses 8 different risks across two broad categories: risk to third parties, and risk to application developers and end users. Compared to previous work, we add new areas focused on offensive security capabilities: automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations. In this paper we discuss applying these benchmarks to the Llama 3 models and a suite of contemporaneous state-of-the-art LLMs, enabling us to contextualize risks both with and without mitigations in place.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 9 of its 16 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

9 of 16 references.

References · 9

  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2023Large Language Modelscited by 774
  3. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  4. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  5. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  6. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  7. 2023Spear Phishing With Large Language Modelscited by 49
  8. 2024Lateral Phishing With Large Language Models: A Large Organization Comparative Studycited by 12
  9. 2024eyeballvul: a future-proof benchmark for vulnerability detection in the wildcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 25; 2026: 25.

56 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  2. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  3. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  4. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  5. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  6. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  7. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  8. 2025Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networkscited by 17
  9. 2025PurpCode: Reasoning for Safer Code Generationcited by 17
  10. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  11. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 15
  12. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
38 more
  1. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 13
  2. 2025OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilitiescited by 12
  3. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  4. 2026Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenderscited by 10
  5. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysiscited by 9
  6. 2025LLMSecConfig: An LLM-Based Approach for Fixing Software Container Misconfigurationscited by 9
  7. 2025Activated LoRA: Fine-tuned LLMs for Intrinsicscited by 9
  8. 2025RedTeamLLM: an Agentic AI framework for offensive securitycited by 6
  9. 2025Concept-Level Explainability for Auditing & Steering LLM Responsescited by 6
  10. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  11. 2025Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitationcited by 4
  12. 2025aiXamine: Simplified LLM Safety and Securitycited by 4
  13. 2024Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critiquecited by 3
  14. 2025The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Modelscited by 2
  15. 2025Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generationcited by 2
  16. 2026AI Agents Enable Adaptive Computer Wormscited by 2
  17. 2026COPA: Continual Preference Optimization for Adaptive Prompt Injection Defensecited by 2
  18. 2024Countering Autonomous Cyber Threatscited by 1
  19. 2025What Makes an Evaluation Useful? Common Pitfalls and Best Practicescited by 1
  20. 2025ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?cited by 1
  21. 2025EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programmingcited by 1
  22. 2025Learning to Align Human Code Preferencescited by 1
  23. 2026RedSage: A Cybersecurity Generalist LLMcited by 1
  24. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  25. 2026VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detectioncited by 1
  26. 2026AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomescited by 1
  27. 2026CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environmentscited by 1
  28. 2026CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledgecited by 1
  29. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  30. 2026Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysiscited by 1
  31. 2026Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysiscited by 1
  32. 2026SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Responsecited by 1
  33. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  34. 2026Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agentscited by 1
  35. 2026CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generationcited by 1
  36. 2026Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contextscited by 0
  37. 2026Risk Reporting for Developers' Internal AI Model Usecited by 0
  38. 2026SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimizationcited by 0

50 most cited of 56 All 56 (JSON)