# Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis

arXiv: https://arxiv.org/abs/2502.01853 · HTML: https://mlsanity.com/paper/2502.01853/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [How Secure is Code Generated by ChatGPT?](https://mlsanity.com/paper/2304.09655/) (2023)
- [Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT](https://mlsanity.com/paper/2304.10778/) (2023)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations](https://mlsanity.com/paper/2303.09384/) (2023)
- [Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities](https://mlsanity.com/paper/2311.16169/) (2023)
- [A large-scale study on research code quality and execution](https://mlsanity.com/paper/2103.12793/) (2021)
- [A User-centered Security Evaluation of Copilot](https://mlsanity.com/paper/2308.06587/) (2023)
- [A Critical Comparison on Six Static Analysis Tools: Detection, Agreement, and Precision](https://mlsanity.com/paper/2101.08832/) (2021)
- [An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability](https://mlsanity.com/paper/2007.12520/) (2020)
- [Exploring the Limits of ChatGPT in Software Security Applications](https://mlsanity.com/paper/2312.05275/) (2023)
- [A case study on the transformative potential of AI in software engineering on LeetCode and ChatGPT](https://mlsanity.com/paper/2501.03639/) (2025)

## Built on by (most cited first, 19 of 19)

- [Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects](https://mlsanity.com/paper/2511.04427/) (2025)
- [A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code](https://mlsanity.com/paper/2508.18106/) (2025)
- [AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms](https://mlsanity.com/paper/2602.09464/) (2026)
- [PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection](https://mlsanity.com/paper/2605.24171/) (2026)
- [Enhancing Reliability in LLM-Based Secure Code Generation](https://mlsanity.com/paper/2605.24300/) (2026)
- [Vulcan: Instance-specialized, Verifiable Systems Heuristics Through LLM-driven Search](https://mlsanity.com/paper/2512.25065/) (2025)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods](https://mlsanity.com/paper/2605.24298/) (2026)
- [Multi-task LLMs for Bug Classification: Efficient Inference with Auxiliary Decoding Heads](https://mlsanity.com/paper/2606.09956/) (2026)
- [CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2609.02774/) (2026)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [LLMs + Security = Trouble](https://mlsanity.com/paper/2602.08422/) (2026)
- [A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Development](https://mlsanity.com/paper/2604.17763/) (2026)
- [Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code](https://mlsanity.com/paper/2607.09902/) (2026)
- [Quantize with Confidence? An Empirical Study of Quantization for Code Generation](https://mlsanity.com/paper/2607.14181/) (2026)
- [Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality](https://mlsanity.com/paper/2607.14816/) (2026)
- [What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code](https://mlsanity.com/paper/2609.12708/) (2026)
- [Security in LLM-Assisted Coding: A Systematic Literature Review](https://mlsanity.com/paper/2610.15482/) (2026)

## Abstract (arXiv)

Artificial Intelligence (AI)-driven code generation tools are increasingly used throughout the software development lifecycle to accelerate coding tasks. However, the security of AI-generated code using Large Language Models (LLMs) remains underexplored, with studies revealing various risks and weaknesses. This paper analyzes the security of code generated by LLMs across different programming languages. We introduce a dataset of 200 tasks grouped into six categories to evaluate the performance of LLMs in generating secure and maintainable code. Our research shows that while LLMs can automate code creation, their security effectiveness varies by language. Many models fail to utilize modern security features in recent compiler and toolkit updates, such as Java 17. Moreover, outdated methods are still commonly used, particularly in C++. This highlights the need for advancing LLMs to enhance security and quality while incorporating emerging best practices in programming languages.