# Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects

arXiv: https://arxiv.org/abs/2406.17910 · HTML: https://mlsanity.com/paper/2406.17910/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 7 of 7)

- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming](https://mlsanity.com/paper/2111.07875/) (2021)

## Built on by (most cited first, 18 of 18)

- [Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects](https://mlsanity.com/paper/2511.04427/) (2025)
- [The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmap](https://mlsanity.com/paper/2412.06512/) (2024)
- [CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning](https://mlsanity.com/paper/2504.14119/) (2025)
- [We Should Evaluate Real-World Impact](https://mlsanity.com/paper/2507.05973/) (2025)
- [CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding](https://mlsanity.com/paper/2512.04111/) (2025)
- [AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development](https://mlsanity.com/paper/2601.13597/) (2026)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [DISCO: Disentangled Communication Steering for Large Language Models](https://mlsanity.com/paper/2509.16820/) (2025)
- [AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software](https://mlsanity.com/paper/2512.18567/) (2025)
- [Accountable Agents in Software Engineering: An Analysis of Terms of Service and a Research Roadmap](https://mlsanity.com/paper/2605.04532/) (2026)
- [Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach](https://mlsanity.com/paper/2508.05693/) (2025)
- [Disrupting Test Development with AI Assistants](https://mlsanity.com/paper/2411.02328/) (2024)
- [Code4MeV2: a Research-oriented Code-completion Platform](https://mlsanity.com/paper/2510.03755/) (2025)
- [The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol](https://mlsanity.com/paper/2606.03907/) (2026)
- [Physically-Relevant Information Learning in High-Dimensional Time-Derivatives Spaces](https://mlsanity.com/paper/2607.05127/) (2026)
- [Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code](https://mlsanity.com/paper/2607.09902/) (2026)
- [Dynamic Adaptation of the LLM Context for Generating Routines with Coupled Semantics](https://mlsanity.com/paper/2609.04570/) (2026)
- [LangSelect: Cost-Aware Target-Language Routing for LLM Code Generation](https://mlsanity.com/paper/2609.18959/) (2026)

## Abstract (arXiv)

Generative AI technologies promise to transform the product development lifecycle. This study evaluates the efficiency gains, areas for improvement, and emerging challenges of using GitHub Copilot, an AI-powered coding assistant. We identified 15 software development tasks and assessed Copilot's benefits through real-world projects on large proprietary code bases. Our findings indicate significant reductions in developer toil, with up to 50% time saved in code documentation and autocompletion, and 30-40% in repetitive coding tasks, unit test generation, debugging, and pair programming. However, Copilot struggles with complex tasks, large functions, multiple files, and proprietary contexts, particularly with C/C++ code. We project a 33-36% time reduction for coding-related tasks in a cloud-first software development lifecycle. This study aims to quantify productivity improvements, identify underperforming scenarios, examine practical benefits and challenges, investigate performance variations across programming languages, and discuss emerging issues related to code quality, security, and developer experience.