# Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions

arXiv: https://arxiv.org/abs/2108.09293 · HTML: https://mlsanity.com/paper/2108.09293/

## Key facts

- Cited by 431 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 39 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [DAVE: Deriving Automatically Verilog from English](https://mlsanity.com/paper/2009.01026/) (2020)
- [Machine Translation from Natural Language to Code using Long-Short Term Memory](https://mlsanity.com/paper/1910.11471/) (2019)

## Built on by (most cited first, 30 of 431)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Who Wrote this Code? Watermarking for Code Generation](https://mlsanity.com/paper/2305.15060/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://mlsanity.com/paper/2312.04724/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [RedCode: Risky Code Execution and Generation Benchmark for Code Agents](https://mlsanity.com/paper/2411.07781/) (2024)
- [Formal Mathematical Reasoning: A New Frontier in AI](https://mlsanity.com/paper/2412.16075/) (2024)
- [A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems](https://mlsanity.com/paper/2402.18649/) (2024)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Clover: Closed-Loop Verifiable Code Generation](https://mlsanity.com/paper/2310.17807/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://mlsanity.com/paper/2406.10279/) (2024)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations](https://mlsanity.com/paper/2303.09384/) (2023)

## Abstract (arXiv)

There is burgeoning interest in designing AI-based systems to assist humans in designing computing systems, including tools that automatically generate computer code. The most notable of these comes in the form of the first self-described `AI pair programmer', GitHub Copilot, a language model trained over open-source GitHub code. However, code often contains bugs - and so, given the vast quantity of unvetted code that Copilot has processed, it is certain that the language model will have learned from exploitable, buggy code. This raises concerns on the security of Copilot's code contributions. In this work, we systematically investigate the prevalence and conditions that can cause GitHub Copilot to recommend insecure code. To perform this analysis we prompt Copilot to generate code in scenarios relevant to high-risk CWEs (e.g. those from MITRE's "Top 25" list). We explore Copilot's performance on three distinct code generation axes -- examining how it performs given diversity of weaknesses, diversity of prompts, and diversity of domains. In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable.