# A User-centered Security Evaluation of Copilot

arXiv: https://arxiv.org/abs/2308.06587 · HTML: https://mlsanity.com/paper/2308.06587/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2023.

## Built on by (most cited first, 15 of 15)

- [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026)
- [Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis](https://mlsanity.com/paper/2502.01853/) (2025)
- [Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios](https://mlsanity.com/paper/2410.12468/) (2024)
- [SandboxEval: Towards Securing Test Environment for Untrusted Code](https://mlsanity.com/paper/2504.00018/) (2025)
- [Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity](https://mlsanity.com/paper/2601.18345/) (2026)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation](https://mlsanity.com/paper/2604.09089/) (2026)
- [Enhancing Reliability in LLM-Based Secure Code Generation](https://mlsanity.com/paper/2605.24300/) (2026)
- [DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents](https://mlsanity.com/paper/2511.20709/) (2025)
- [A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Development](https://mlsanity.com/paper/2604.17763/) (2026)
- [Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study](https://mlsanity.com/paper/2606.25973/) (2026)
- [Functional and Secure Code Generation with Task Vectors](https://mlsanity.com/paper/2607.07881/) (2026)
- [Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Study](https://mlsanity.com/paper/2607.11348/) (2026)
- [Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality](https://mlsanity.com/paper/2607.14816/) (2026)
- [Beware EviLLM: Enabling Vulnerability Injection via Large Language Models](https://mlsanity.com/paper/2610.03857/) (2026)

## Abstract (arXiv)

Code generation tools driven by artificial intelligence have recently become more popular due to advancements in deep learning and natural language processing that have increased their capabilities. The proliferation of these tools may be a double-edged sword because while they can increase developer productivity by making it easier to write code, research has shown that they can also generate insecure code. In this paper, we perform a user-centered evaluation GitHub's Copilot to better understand its strengths and weaknesses with respect to code security. We conduct a user study where participants solve programming problems (with and without Copilot assistance) that have potentially vulnerable solutions. The main goal of the user study is to determine how the use of Copilot affects participants' security performance. In our set of participants (n=25), we find that access to Copilot accompanies a more secure solution when tackling harder problems. For the easier problem, we observe no effect of Copilot access on the security of solutions. We also observe no disproportionate impact of Copilot use on particular kinds of vulnerabilities. Our results indicate that there are potential security benefits to using Copilot, but more research is warranted on the effects of the use of code generation tools on technically complex problems with security requirements.