mlsanityData to 7 Oct 2026

Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models

arXiv 2312.04724 ↗First posted 7 Dec 2023cs.CR cs.LGcited by 130

Not on the map. Why?

Cited by 33 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants. As what we believe to be the most extensive unified cybersecurity safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs in two crucial security domains: their propensity to generate insecure code and their level of compliance when asked to assist in cyberattacks. Through a case study involving seven models from the Llama 2, Code Llama, and OpenAI GPT large language model families, CyberSecEval effectively pinpointed key cybersecurity risks. More importantly, it offered practical insights for refining these models. A significant observation from the study was the tendency of more advanced models to suggest insecure code, highlighting the critical need for integrating security considerations in the development of sophisticated LLMs. CyberSecEval, with its automated test case generation and evaluation pipeline covers a broad scope and equips LLM designers and researchers with a tool to broadly measure and enhance the cybersecurity safety properties of LLMs, contributing to the development of more secure AI systems.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 11 of its 19 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

11 of 19 references.

References · 11

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2023StarCoder: may the source be with you!cited by 790
  5. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  6. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 87
  7. 2023How Secure is Code Generated by ChatGPT?cited by 77
  8. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  9. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  10. 2023The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verificationcited by 32
  11. 2023AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 45; 2025: 46; 2026: 39.

130 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2025Humanity's Last Examcited by 467
  3. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  4. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  5. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  6. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  7. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 87
  8. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  9. 2024Visibility into AI Agentscited by 77
  10. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  11. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  12. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
38 more
  1. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 58
  2. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  3. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 56
  4. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 55
  5. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  6. 2024Sabotage Evaluations for Frontier Modelscited by 45
  7. 2025Infrastructure for AI Agentscited by 40
  8. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  9. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 36
  10. 2024Constrained Decoding for Secure Code Generationcited by 35
  11. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  12. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 26
  13. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  14. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  15. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  16. 2024CodeMirage: Hallucinations in Code Generated by Large Language Modelscited by 23
  17. 2024Locking Down the Finetuned LLMs Safetycited by 22
  18. 2024PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedbackcited by 22
  19. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  20. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  21. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  22. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  23. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 15
  24. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  25. 2024HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Datacited by 14
  26. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositoriescited by 14
  27. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 13
  28. 2026Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarioscited by 13
  29. 2025Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Modelscited by 12
  30. 2025OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilitiescited by 12
  31. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 12
  32. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Levelcited by 11
  33. 2025Training Language Models to Generate Quality Code with Program Analysis Feedbackcited by 11
  34. 2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
  35. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 11
  36. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  37. 2024How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studiescited by 9
  38. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysiscited by 9

50 most cited of 130 All 130 (JSON)