# Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study

arXiv: https://arxiv.org/abs/2310.02059 · HTML: https://mlsanity.com/paper/2310.02059/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2023.

## Built on by (most cited first, 30 of 38)

- [SWE-chat: Coding Agent Interactions From Real Users in the Wild](https://mlsanity.com/paper/2604.20779/) (2026)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis](https://mlsanity.com/paper/2505.20630/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [On LLMs' Internal Representation of Code Correctness](https://mlsanity.com/paper/2512.07404/) (2025)
- [Large Language Model Unlearning for Source Code](https://mlsanity.com/paper/2506.17125/) (2025)
- [AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software](https://mlsanity.com/paper/2512.18567/) (2025)
- [Agentic AI in Industry: Adoption Level and Deployment Barriers](https://mlsanity.com/paper/2605.14675/) (2026)
- [What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants](https://mlsanity.com/paper/2605.30777/) (2026)
- [Demystifying Issues, Causes and Solutions in LLM Open-Source Projects](https://mlsanity.com/paper/2409.16559/) (2024)
- [Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies](https://mlsanity.com/paper/2510.22944/) (2025)
- [CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agents](https://mlsanity.com/paper/2512.21250/) (2025)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [An Empirical Study of Vulnerabilities in Python Packages and Their Detection](https://mlsanity.com/paper/2509.04260/) (2025)
- [Enhancing Reliability in LLM-Based Secure Code Generation](https://mlsanity.com/paper/2605.24300/) (2026)
- [DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents](https://mlsanity.com/paper/2511.20709/) (2025)
- [Just-in-Time Catching Test Generation at Meta](https://mlsanity.com/paper/2601.22832/) (2026)
- [RocqSmith: Can Automatic Optimization Forge Better Proof Agents?](https://mlsanity.com/paper/2602.05762/) (2026)
- [Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions](https://mlsanity.com/paper/2605.06279/) (2026)
- [SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization](https://mlsanity.com/paper/2605.08382/) (2026)
- [On the Robustness of LLMs' Internal Representation of Code Correctness](https://mlsanity.com/paper/2608.08266/) (2026)
- [Designing Empirical Studies on LLM-Based Code Generation: Towards a Reference Framework](https://mlsanity.com/paper/2510.03862/) (2025)
- [EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization](https://mlsanity.com/paper/2602.22368/) (2026)
- [Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning](https://mlsanity.com/paper/2603.17174/) (2026)
- [A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Development](https://mlsanity.com/paper/2604.17763/) (2026)
- [Quality and Security Signals in AI-Generated Python Refactoring Pull Requests](https://mlsanity.com/paper/2605.21453/) (2026)
- [Short-Term Gain, Long-Term Fragility: AI Labor Substitution and the Erosion of Sustainable Capability](https://mlsanity.com/paper/2605.27399/) (2026)
- [Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study](https://mlsanity.com/paper/2606.25973/) (2026)
- [Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Study](https://mlsanity.com/paper/2607.11348/) (2026)
- [The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics](https://mlsanity.com/paper/2607.13077/) (2026)

## Abstract (arXiv)

Modern code generation tools utilizing AI models like Large Language Models (LLMs) have gained increased popularity due to their ability to produce functional code. However, their usage presents security challenges, often resulting in insecure code merging into the code base. Thus, evaluating the quality of generated code, especially its security, is crucial. While prior research explored various aspects of code generation, the focus on security has been limited, mostly examining code produced in controlled environments rather than open source development scenarios. To address this gap, we conducted an empirical study, analyzing code snippets generated by GitHub Copilot and two other AI code generation tools (i.e., CodeWhisperer and Codeium) from GitHub projects. Our analysis identified 733 snippets, revealing a high likelihood of security weaknesses, with 29.5% of Python and 24.2% of JavaScript snippets affected. These issues span 43 Common Weakness Enumeration (CWE) categories, including significant ones like CWE-330: Use of Insufficiently Random Values, CWE-94: Improper Control of Generation of Code, and CWE-79: Cross-site Scripting. Notably, eight of those CWEs are among the 2023 CWE Top-25, highlighting their severity. We further examined using Copilot Chat to fix security issues in Copilot-generated code by providing Copilot Chat with warning messages from the static analysis tools, and up to 55.5% of the security issues can be fixed. We finally provide the suggestions for mitigating security issues in generated code.