mlsanityData to 7 Oct 2026

GitHub Copilot AI pair programmer: Asset or Liability?

arXiv 2206.15331 ↗First posted 30 Jun 2022cs.SE cs.LGcited by 150

Not on the map. Why?

Abstract (arXiv)

Automatic program synthesis is a long-lasting dream in software engineering. Recently, a promising Deep Learning (DL) based solution, called Copilot, has been proposed by OpenAI and Microsoft as an industrial product. Although some studies evaluate the correctness of Copilot solutions and report its issues, more empirical evaluations are necessary to understand how developers can benefit from it effectively. In this paper, we study the capabilities of Copilot in two different programming tasks: (i) generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems, and (ii) comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks. For the former, we assess the performance and functionality of Copilot in solving selected fundamental problems in computer science, like sorting and implementing data structures. In the latter, a dataset of programming problems with human-provided solutions is used. The results show that Copilot is capable of providing solutions for almost all fundamental algorithmic problems, however, some solutions are buggy and non-reproducible. Moreover, Copilot has some difficulties in combining multiple methods to generate a solution. Comparing Copilot to humans, our results show that the correct ratio of humans' solutions is greater than Copilot's suggestions, while the buggy solutions generated by Copilot require less effort to be repaired.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 18 of its 66 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

18 of 66 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  7. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  8. 2022Productivity Assessment of Neural Code Completioncited by 97
  9. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  10. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
  11. 2019Does BLEU Score Work for Code Migration?cited by 49
  12. 2021Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programmingcited by 24
6 more
  1. 2016Automated Clustering and Program Repair for Introductory Programming Assignmentscited by 21
  2. 2021Solving Linear Algebra by Program Synthesiscited by 14
  3. 2021Solving Probability and Statistics Problems by Program Synthesiscited by 9
  4. 2021Recent Developments in Program Synthesis with Evolutionary Algorithmscited by 5
  5. 2019Machine Translation from Natural Language to Code using Long-Short Term Memorycited by 4
  6. 2021Readability and Understandability Scores for Snippet Assessment: an Exploratory Studycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 33; 2024: 59; 2025: 38; 2026: 17.

150 papers.

Sort:

By area:

  1. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  4. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 97
  5. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  6. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  7. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  8. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 88
  9. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 85
  10. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 81
  11. 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
  12. 2024Building Machines that Learn and Think with Peoplecited by 73
38 more
  1. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  2. 2024MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agentscited by 64
  3. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  4. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 57
  5. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHubcited by 53
  6. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  7. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 50
  8. 2023Computing Education in the Era of Generative AIcited by 50
  9. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 45
  10. 2023A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verificationcited by 42
  11. 2023Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questionscited by 42
  12. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  13. 2024Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completioncited by 40
  14. 2024Generative Artificial Intelligence: A Systematic Review and Applicationscited by 37
  15. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
  16. 2023SALLM: Security Assessment of Generated Codecited by 35
  17. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  18. 2025Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Supportcited by 33
  19. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  20. 2023Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Casescited by 32
  21. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  22. 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
  23. 2023How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debuggingcited by 29
  24. 2024The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiringcited by 29
  25. 2024An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Projectcited by 28
  26. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  27. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 27
  28. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  29. 2023Transforming Sentiment Analysis in the Financial Domain with ChatGPTcited by 25
  30. 2024Qiskit Code Assistant: Training LLMs for generating Quantum Computing Codecited by 24
  31. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  32. 2024Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chatcited by 23
  33. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  34. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  35. 2024XAI for All: Can Large Language Models Simplify Explainable AI?cited by 21
  36. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
  37. 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
  38. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20

50 most cited of 150 All 150 (JSON)