mlsanityData to 7 Oct 2026

LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations

arXiv 2303.09384 ↗First posted 16 Mar 2023cs.SE cs.IR cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) like Codex are powerful tools for performing code completion and code generation tasks as they are trained on billions of lines of code from publicly available sources. Moreover, these models are capable of generating code snippets from Natural Language (NL) descriptions by learning languages and programming practices from public GitHub repositories. Although LLMs promise an effortless NL-driven deployment of software applications, the security of the code they generate has not been extensively investigated nor documented. In this work, we present LLMSecEval, a dataset containing 150 NL prompts that can be leveraged for assessing the security performance of such models. Such prompts are NL descriptions of code snippets prone to various security vulnerabilities listed in MITRE's Top 25 Common Weakness Enumeration (CWE) ranking. Each prompt in our dataset comes with a secure implementation example to facilitate comparative evaluations against code produced by LLMs. As a practical application, we show how LLMSecEval can be used for evaluating the security of snippets automatically generated from NL descriptions.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 10 of its 19 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

10 of 19 references.

References · 10

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  4. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  5. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  6. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  7. 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
  8. 2019A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Softwarecited by 28
  9. 2020Java Cryptography Uses in the Wildcited by 3
  10. 2022GitHub Considered Harmful? Analyzing Open-Source Projects for the Automatic Generation of Cryptographic API Call Sequencescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 13; 2025: 18; 2026: 20.

58 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  3. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  4. 2023SALLM: Security Assessment of Generated Codecited by 35
  5. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  7. 2023Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attackscited by 29
  8. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 26
  9. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  10. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  11. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  12. 2024PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedbackcited by 22
38 more
  1. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  2. 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 18
  3. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 16
  4. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  5. 2025Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Modelscited by 12
  6. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 11
  7. 2025Rethinking the Evaluation of Secure Code Generationcited by 10
  8. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  9. 2025Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferencescited by 10
  10. 2023Using ChatGPT as a Static Application Security Testing Toolcited by 8
  11. 2024INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulnesscited by 7
  12. 2023Automating the Correctness Assessment of AI-generated Code for Security Contextscited by 6
  13. 2025Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspectivecited by 6
  14. 2025CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrievalcited by 6
  15. 2025QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedbackcited by 6
  16. 2023Towards Trustworthy AI Software Development Assistancecited by 4
  17. 2024FATS: A Prompt Injection Attack Utilizing Feign Security Agents with Deceptive Few-shots Learningcited by 4
  18. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  19. 2026Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Modelcited by 4
  20. 2023Can LLMs Patch Security Issues?cited by 3
  21. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  22. 2025Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategiescited by 3
  23. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Modelscited by 2
  24. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 2
  25. 2025Secure Code Generation at Scale with Reflexioncited by 1
  26. 2026Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Softwarecited by 1
  27. 2026SecCodeBench-V2 Technical Reportcited by 1
  28. 2026VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detectioncited by 1
  29. 2026SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalizationcited by 1
  30. 2026Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Codecited by 1
  31. 2026Social Bias in LLM-Generated Code: Benchmark and Mitigationcited by 1
  32. 2026Constrained Code Generation with Discrete Diffusioncited by 1
  33. 2026Enhancing Reliability in LLM-Based Secure Code Generationcited by 1
  34. 2024Dynamic Code Orchestration: Harnessing the Power of Large Language Models for Adaptive Script Executioncited by 0
  35. 2025Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software developmentcited by 0
  36. 2025Assessing the Software Security Comprehension of Large Language Modelscited by 0
  37. 2026Inference-Time Safety For Code LLMs Via Retrieval-Augmented Revisioncited by 0
  38. 2026Seclens: Role-specific Evaluation of LLM's for security vulnerablity detectioncited by 0

50 most cited of 58 All 58 (JSON)