# From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security

arXiv: https://arxiv.org/abs/2412.15004 · HTML: https://mlsanity.com/paper/2412.15004/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 77 of its 159 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 30 of 77)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [How Secure is Code Generated by ChatGPT?](https://mlsanity.com/paper/2304.09655/) (2023)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [TrojanPuzzle: Covertly Poisoning Code-Suggestion Models](https://mlsanity.com/paper/2301.02344/) (2023)
- [Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities](https://mlsanity.com/paper/2311.16169/) (2023)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- [CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models](https://mlsanity.com/paper/2302.04012/) (2023)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)

## Built on by (most cited first, 10 of 10)

- [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systems](https://mlsanity.com/paper/2512.23132/) (2025)
- [PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection](https://mlsanity.com/paper/2605.24171/) (2026)
- [Secure Code Generation at Scale with Reflexion](https://mlsanity.com/paper/2511.03898/) (2025)
- [Constrained Code Generation with Discrete Diffusion](https://mlsanity.com/paper/2605.16829/) (2026)
- [Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)](https://mlsanity.com/paper/2605.20351/) (2026)
- [Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda](https://mlsanity.com/paper/2608.21107/) (2026)
- [Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen](https://mlsanity.com/paper/2609.26749/) (2026)
- [Security in LLM-Assisted Coding: A Systematic Literature Review](https://mlsanity.com/paper/2610.15482/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have emerged as powerful tools for automating programming tasks, including security-related ones. However, they can also introduce vulnerabilities during code generation, fail to detect existing vulnerabilities, or report nonexistent ones. This systematic literature review investigates the security benefits and drawbacks of using LLMs for code-related tasks. In particular, it focuses on the types of vulnerabilities introduced by LLMs when generating code. Moreover, it analyzes the capabilities of LLMs to detect and fix vulnerabilities, and examines how prompting strategies impact these tasks. Finally, it examines how data poisoning attacks impact LLMs performance in the aforementioned tasks.