mlsanityData to 14 Oct 2026

CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

arXiv 2404.13161 ↗First posted 19 Apr 2024cs.CR cs.LGcited by 89

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present BenchmarkName, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state-of-the-art (SOTA) LLMs, including GPT-4, Mistral, Meta Llama 3 70B-Instruct, and Code Llama. Our results show that conditioning away risk of attack remains an unsolved problem; for example, all tested models showed between 26% and 41% successful prompt injection tests. We further introduce the safety-utility tradeoff: conditioning an LLM to reject unsafe prompts can cause the LLM to falsely reject answering benign prompts, which lowers utility. We propose quantifying this tradeoff using False Refusal Rate (FRR). As an illustration, we introduce a novel test set to quantify FRR for cyberattack helpfulness risk. We find many LLMs able to successfully comply with "borderline" benign requests while still rejecting most unsafe requests. Finally, we quantify the utility of LLMs for automating a core cybersecurity task, that of exploiting software vulnerabilities. This is important because the offensive capabilities of LLMs are of intense interest; we quantify this by creating novel test sets for four representative problems. We find that models with coding capabilities perform better than those without, but that further work is needed for LLMs to become proficient at exploit generation. Our code is open source and can be used to evaluate other LLMs.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 6 of its 11 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

6 of 11 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb

Other references · 5

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  3. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  4. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 59
  5. 2023SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Securitycited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 33; 2026: 33.

89 papers.

Sort:

By area:

  1. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  2. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 96
  3. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
  4. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 75
  5. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  6. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
  7. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 52
  8. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 45
  9. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 31
  10. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 22
  11. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  12. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 17
38 more
  1. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 16
  2. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 16
  3. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 15
  4. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  6. 2026IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMscited by 15
  7. 2024Hacking CTFs with Plain Agentscited by 14
  8. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 14
  9. 2025CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Executioncited by 14
  10. 2024Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilitiescited by 12
  11. 2025OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilitiescited by 12
  12. 2025Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agentscited by 12
  13. 2024Badllama 3: removing safety finetuning from Llama 3 in minutescited by 11
  14. 2025A Contemporary Survey of Large Language Model Assisted Program Analysiscited by 11
  15. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysiscited by 10
  16. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 10
  17. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  18. 2025LLMSecConfig: An LLM-Based Approach for Fixing Software Container Misconfigurationscited by 9
  19. 2025RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agentscited by 9
  20. 2024INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulnesscited by 8
  21. 2024Efficient Federated Intrusion Detection in 5G ecosystem using optimized BERT-based modelcited by 8
  22. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 8
  23. 2025SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysiscited by 8
  24. 2024eyeballvul: a future-proof benchmark for vulnerability detection in the wildcited by 6
  25. 2024CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researchercited by 6
  26. 2025Logic layer Prompt Control Injection (LPCI): A Novel Security Vulnerability Class in Agentic Systemscited by 6
  27. 2024Are You Human? An Adversarial Benchmark to Expose LLMscited by 5
  28. 2024Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidencecited by 5
  29. 2025Evaluating AI cyber capabilities with crowdsourced elicitationcited by 5
  30. 2026HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generationcited by 5
  31. 2026CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agentscited by 5
  32. 2026Agent Memory Is a Surface for Endogenous Authorization Launderingcited by 5
  33. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  34. 2024What AI evaluations for preventing catastrophic risks can and cannot docited by 4
  35. 2025DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environmentscited by 4
  36. 2026Towards Optimal Agentic Architectures for Offensive Security Taskscited by 4
  37. 2024A Misleading Gallery of Fluid Motion by Generative Artificial Intelligencecited by 3
  38. 2024Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critiquecited by 3

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data
  2. 2025Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security
  3. 2024Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications
  4. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  5. 2025A Critical Evaluation of Defenses against Prompt Injection Attacks
  6. 2025LLM-Safety Evaluations Lack Robustness
  7. 2025CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuning
  8. 2025Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques