# Security of LLM-generated Code: A Comparative Analysis

arXiv: https://arxiv.org/abs/2605.23091 · HTML: https://mlsanity.com/paper/2605.23091/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 77 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 21 of 21)

- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [How Secure is Code Generated by ChatGPT?](https://mlsanity.com/paper/2304.09655/) (2023)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations](https://mlsanity.com/paper/2303.09384/) (2023)
- [TrojanPuzzle: Covertly Poisoning Code-Suggestion Models](https://mlsanity.com/paper/2301.02344/) (2023)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [Prompt-Enhanced Software Vulnerability Detection Using ChatGPT](https://mlsanity.com/paper/2308.12697/) (2023)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [LeDex: Training LLMs to Better Self-Debug and Explain Code](https://mlsanity.com/paper/2405.18649/) (2024)
- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks](https://mlsanity.com/paper/2308.04451/) (2023)
- [Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning](https://mlsanity.com/paper/2406.03718/) (2024)
- [Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Study](https://mlsanity.com/paper/2311.11177/) (2023)
- [Exploring ChatGPT's Capabilities on Vulnerability Management](https://mlsanity.com/paper/2311.06530/) (2023)
- [SourceFinder: Finding Malware Source-Code from Publicly Available Repositories](https://mlsanity.com/paper/2005.14311/) (2020)
- [Can Large Language Models Find And Fix Vulnerable Software?](https://mlsanity.com/paper/2308.10345/) (2023)
- [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://mlsanity.com/paper/2506.23034/) (2025)

## Abstract (arXiv)

The majority of software developers use or are planning to use Artificial Intelligence (AI) tools in their development processes. Their top reasons include improving productivity and faster learning. In fact, Large Language Model (LLM)-generated code is currently in production, including in major tech companies. However, concerns were raised about the risks associated with the use of AI tools to generate code. In this paper, we focus our attention on the risks to software security. We empirically evaluate the security of code generated by seven popular LLMs. We build upon previous work to mimic the behaviours of developers when using LLMs to generate code. Our results show that all seven LLMs that we have evaluated generate code that contains vulnerabilities, the majority of which are of critical or high severity.