# Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Study

arXiv: https://arxiv.org/abs/2311.11177 · HTML: https://mlsanity.com/paper/2311.11177/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2023.

## Built on by (most cited first, 26 of 26)

- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios](https://mlsanity.com/paper/2410.12468/) (2024)
- ["Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors](https://mlsanity.com/paper/2509.22040/) (2025)
- [Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios](https://mlsanity.com/paper/2503.12374/) (2025)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [SandboxEval: Towards Securing Test Environment for Untrusted Code](https://mlsanity.com/paper/2504.00018/) (2025)
- [RESCUE: Retrieval Augmented Secure Code Generation](https://mlsanity.com/paper/2510.18204/) (2025)
- [On LLMs' Internal Representation of Code Correctness](https://mlsanity.com/paper/2512.07404/) (2025)
- [Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassian](https://mlsanity.com/paper/2501.11264/) (2025)
- [SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows](https://mlsanity.com/paper/2506.07313/) (2025)
- [TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models](https://mlsanity.com/paper/2407.21227/) (2024)
- [Model Science: getting serious about verification, explanation and control of AI systems](https://mlsanity.com/paper/2508.20040/) (2025)
- [Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies](https://mlsanity.com/paper/2510.22944/) (2025)
- [A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models](https://mlsanity.com/paper/2409.15228/) (2024)
- [Developer Perspectives on Licensing and Copyright Issues Arising from Generative AI for Software Development](https://mlsanity.com/paper/2411.10877/) (2024)
- [DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation](https://mlsanity.com/paper/2604.09089/) (2026)
- [DeepCodeProbe: Towards Understanding What Models Trained on Code Learn](https://mlsanity.com/paper/2407.08890/) (2024)
- [The Role of GitHub Copilot on Software Development: A Perspective on Productivity, Security, Best Practices and Future Directions](https://mlsanity.com/paper/2502.13199/) (2025)
- [On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practices](https://mlsanity.com/paper/2504.16485/) (2025)
- [Risks of AI-driven product development and strategies for their mitigation](https://mlsanity.com/paper/2506.00047/) (2025)
- [DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents](https://mlsanity.com/paper/2511.20709/) (2025)
- [SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization](https://mlsanity.com/paper/2605.08382/) (2026)
- [How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?](https://mlsanity.com/paper/2512.15468/) (2025)
- [Security of LLM-generated Code: A Comparative Analysis](https://mlsanity.com/paper/2605.23091/) (2026)
- [Beware EviLLM: Enabling Vulnerability Injection via Large Language Models](https://mlsanity.com/paper/2610.03857/) (2026)
- [Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair](https://mlsanity.com/paper/2610.06163/) (2026)

## Abstract (arXiv)

AI-powered code generation models have been developing rapidly, allowing developers to expedite code generation and thus improve their productivity. These models are trained on large corpora of code (primarily sourced from public repositories), which may contain bugs and vulnerabilities. Several concerns have been raised about the security of the code generated by these models. Recent studies have investigated security issues in AI-powered code generation tools such as GitHub Copilot and Amazon CodeWhisperer, revealing several security weaknesses in the code generated by these tools. As these tools evolve, it is expected that they will improve their security protocols to prevent the suggestion of insecure code to developers. This paper replicates the study of Pearce et al., which investigated security weaknesses in Copilot and uncovered several weaknesses in the code suggested by Copilot across diverse scenarios and languages (Python, C and Verilog). Our replication examines Copilot security weaknesses using newer versions of Copilot and CodeQL (the security analysis framework). The replication focused on the presence of security vulnerabilities in Python code. Our results indicate that, even with the improvements in newer versions of Copilot, the percentage of vulnerable code suggestions has reduced from 36.54% to 27.25%. Nonetheless, it remains evident that the model still suggests insecure code.