# Practices and Challenges of Using GitHub Copilot: An Empirical Study

arXiv: https://arxiv.org/abs/2303.08733 · HTML: https://mlsanity.com/paper/2303.08733/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2023.

## Built on by (most cited first, 14 of 14)

- [SandboxEval: Towards Securing Test Environment for Untrusted Code](https://mlsanity.com/paper/2504.00018/) (2025)
- [Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis](https://mlsanity.com/paper/2412.14841/) (2024)
- [Evaluating AI-generated code for C++, Fortran, Go, Java, Julia, Matlab, Python, R, and Rust](https://mlsanity.com/paper/2405.13101/) (2024)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [Impact of AI-tooling on the Engineering Workspace](https://mlsanity.com/paper/2406.07683/) (2024)
- [A Certified Robust Watermark For Large Language Models](https://mlsanity.com/paper/2409.19708/) (2024)
- [Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering](https://mlsanity.com/paper/2603.26567/) (2026)
- [An Exploratory Study on LLM-Generated Code and Comments in Code Repositories](https://mlsanity.com/paper/2607.01867/) (2026)
- [Exploring utilization of generative AI for research and education in data-driven materials science](https://mlsanity.com/paper/2504.08817/) (2025)
- [From Specification to Service: Accelerating API-First Development Using Multi-Agent Systems](https://mlsanity.com/paper/2510.19274/) (2025)
- [Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Code](https://mlsanity.com/paper/2601.19264/) (2026)
- ["I Just Need GPT to Refine My Prompts": Rethinking Onboarding and Help-Seeking with Generative 3D Modeling Tools](https://mlsanity.com/paper/2603.29118/) (2026)
- [The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics](https://mlsanity.com/paper/2607.13077/) (2026)
- [Building a research-software catalog with a coding agent: from hackathon prototype to public deployment](https://mlsanity.com/paper/2609.04711/) (2026)

## Abstract (arXiv)

With the advances in machine learning, there is a growing interest in AI-enabled tools for autocompleting source code. GitHub Copilot, also referred to as the "AI Pair Programmer", has been trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch on June 2021. However, little effort has been devoted to understanding the practices and challenges of using Copilot in programming with auto-completed source code. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. More specifically, we searched and manually collected 169 SO posts and 655 GitHub discussions related to the usage of Copilot. We identified the programming languages, IDEs, technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. The results show that when practitioners use Copilot: (1) The major programming languages used with Copilot are JavaScript and Python, (2) the main IDE used with Copilot is Visual Studio Code, (3) the most common used technology with Copilot is Node.js, (4) the leading function implemented by Copilot is data processing, (5) the significant benefit of using Copilot is useful code generation, and (6) the main limitation encountered by practitioners when using Copilot is difficulty of integration. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it. Our study provides empirically grounded foundations and basis for future research on the role of Copilot as an AI pair programmer in software development.