mlsanityData to 7 Oct 2026

Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis

arXiv 2502.01853 ↗First posted 3 Feb 2025cs.CR cs.LG cs.SEcited by 18

Not on the map. Why?

Abstract (arXiv)

Artificial Intelligence (AI)-driven code generation tools are increasingly used throughout the software development lifecycle to accelerate coding tasks. However, the security of AI-generated code using Large Language Models (LLMs) remains underexplored, with studies revealing various risks and weaknesses. This paper analyzes the security of code generated by LLMs across different programming languages. We introduce a dataset of 200 tasks grouped into six categories to evaluate the performance of LLMs in generating secure and maintainable code. Our research shows that while LLMs can automate code creation, their security effectiveness varies by language. Many models fail to utilize modern security features in recent compiler and toolkit updates, such as Java 17. Moreover, outdated methods are still commonly used, particularly in C++. This highlights the need for advancing LLMs to enhance security and quality while incorporating emerging best practices in programming languages.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 23 of its 68 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

23 of 68 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2024Large Language Models: A Surveycited by 506
  7. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  8. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  9. 2020Explainable Deep Learning: A Field Guide for the Uninitiatedcited by 127
  10. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  11. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 94
  12. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 87
11 more
  1. 2023How Secure is Code Generated by ChatGPT?cited by 77
  2. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  3. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
  4. 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 58
  5. 2023Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilitiescited by 58
  6. 2021A large-scale study on research code quality and executioncited by 22
  7. 2023A User-centered Security Evaluation of Copilotcited by 14
  8. 2021A Critical Comparison on Six Static Analysis Tools: Detection, Agreement, and Precisioncited by 9
  9. 2020An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandabilitycited by 7
  10. 2023Exploring the Limits of ChatGPT in Software Security Applicationscited by 5
  11. 2025A case study on the transformative potential of AI in software engineering on LeetCode and ChatGPTcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  2. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 12
  3. 2026AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithmscited by 11
  4. 2025Vulcan: Instance-specialized, Verifiable Systems Heuristics Through LLM-driven Searchcited by 1
  5. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  6. 2026PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detectioncited by 1
  7. 2026Enhancing Reliability in LLM-Based Secure Code Generationcited by 1
  8. 2026Multi-task LLMs for Bug Classification: Efficient Inference with Auxiliary Decoding Headscited by 1
  9. 2026CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generationcited by 1
  10. 2025Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniquescited by 0
  11. 2026LLMs + Security = Troublecited by 0
  12. 2026A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Developmentcited by 0
6 more
  1. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  2. 2026An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methodscited by 0
  3. 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Codecited by 0
  4. 2026Quantize with Confidence? An Empirical Study of Quantization for Code Generationcited by 0
  5. 2026Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Qualitycited by 0
  6. 2026What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Codecited by 0