mlsanityData to 14 Oct 2026

From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security

arXiv 2412.15004 ↗First posted 19 Dec 2024cs.CR cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have emerged as powerful tools for automating programming tasks, including security-related ones. However, they can also introduce vulnerabilities during code generation, fail to detect existing vulnerabilities, or report nonexistent ones. This systematic literature review investigates the security benefits and drawbacks of using LLMs for code-related tasks. In particular, it focuses on the types of vulnerabilities introduced by LLMs when generating code. Moreover, it analyzes the capabilities of LLMs to detect and fix vulnerabilities, and examines how prompting strategies impact these tasks. Finally, it examines how data poisoning attacks impact LLMs performance in the aforementioned tasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 77 of its 159 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

77 of 159 references.

References · 77

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  8. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  9. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  10. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  11. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  12. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
38 more
  1. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 144
  2. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  3. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 131
  4. 2024IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilitiescited by 103
  5. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 97
  6. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
  7. 2019Slither: A Static Analysis Framework For Smart Contractscited by 89
  8. 2023Getting pwn'd by AI: Penetration Testing with Large Language Modelscited by 83
  9. 2023How Secure is Code Generated by ChatGPT?cited by 81
  10. 2024Instruction Tuning for Secure Code Generationcited by 68
  11. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 67
  12. 2023GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysiscited by 65
  13. 2023TrojanPuzzle: Covertly Poisoning Code-Suggestion Modelscited by 60
  14. 2023Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilitiescited by 59
  15. 2021Vulnerability Detection with Fine-grained Interpretationscited by 56
  16. 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 56
  17. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 55
  18. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 52
  19. 2023Prompt-Enhanced Software Vulnerability Detection Using ChatGPTcited by 51
  20. 2023Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectivescited by 48
  21. 2023No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPTcited by 47
  22. 2024Current state of LLM Risks and AI Guardrailscited by 46
  23. 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 45
  24. 2023ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?cited by 44
  25. 2023When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?cited by 42
  26. 2024Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Aheadcited by 42
  27. 2019Getafix: Learning to Fix Bugs Automaticallycited by 41
  28. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 40
  29. 2022Transformer-Based Language Models for Software Vulnerability Detectioncited by 39
  30. 2023SALLM: Security Assessment of Generated Codecited by 39
  31. 2023Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Studycited by 38
  32. 2021Neural Transfer Learning for Repairing Security Vulnerabilities in C Codecited by 36
  33. 2023Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attackscited by 33
  34. 2024Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilitiescited by 31
  35. 2023Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Studycited by 26
  36. 2024LLMs in Web Development: Evaluating LLM-Generated PHP Code Unveiling Vulnerabilities and Limitationscited by 22
  37. 2024A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedbackcited by 22
  38. 2021VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatementscited by 20

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025A Contemporary Survey of Large Language Model Assisted Program Analysiscited by 11
  2. 2025Trust Me, I Know This Function: Hijacking LLM Static Analysis using Biascited by 3
  3. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
  4. 2026PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detectioncited by 2
  5. 2025Secure Code Generation at Scale with Reflexioncited by 1
  6. 2026Constrained Code Generation with Discrete Diffusioncited by 1
  7. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  8. 2026Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agendacited by 0
  9. 2026Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screencited by 0
  10. 2026Security in LLM-Assisted Coding: A Systematic Literature Reviewcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
  2. 2025Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks
  3. 2026An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods
  4. 2025When Code Crosses Borders: A Security-Centric Study of LLM-based Code Translation ↗
  5. 2024Software Vulnerability and Functionality Assessment using LLMs
  6. 2026Enhancing Reliability in LLM-Based Secure Code Generation
  7. 2025Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts
  8. 2025Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security