mlsanityData to 7 Oct 2026

Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming

arXiv 2111.07875 ↗First posted 15 Nov 2021cs.SE cs.LG cs.NEcited by 24

Not on the map. Why?

Abstract (arXiv)

GitHub Copilot, an extension for the Visual Studio Code development environment powered by the large-scale language model Codex, makes automatic program synthesis available for software developers. This model has been extensively studied in the field of deep learning, however, a comparison to genetic programming, which is also known for its performance in automatic program synthesis, has not yet been carried out. In this paper, we evaluate GitHub Copilot on standard program synthesis benchmark problems and compare the achieved results with those from the genetic programming literature. In addition, we discuss the performance of both approaches. We find that the performance of the two approaches on the benchmark problems is quite similar, however, in comparison to GitHub Copilot, the program synthesis approaches based on genetic programming are not yet mature enough to support programmers in practical software development. Genetic programming usually needs a huge amount of expensive hand-labeled training cases and takes too much time to generate solutions. Furthermore, source code generated by genetic programming approaches is often bloated and difficult to understand. For future work on program synthesis with genetic programming, we suggest researchers to focus on improving the execution time, readability, and usability.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 15 of its 39 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

15 of 39 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  10. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  11. 2020SourceFinder: Finding Malware Source-Code from Publicly Available Repositoriescited by 14
  12. 2021PSB2: The Second Program Synthesis Benchmark Suitecited by 9
3 more
  1. 2021Problem-solving benefits of down-sampled lexicase selectioncited by 8
  2. 2021TensorGP -- Genetic Programming Engine in TensorFlowcited by 5
  3. 2021Recent Developments in Program Synthesis with Evolutionary Algorithmscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 12; 2024: 7; 2025: 2; 2026: 1.

24 papers.

Sort:

By area:

  1. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  2. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  3. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  4. 2023Large Language Models and Simple, Stupid Bugscited by 45
  5. 2024Evolving Code with A Large Language Modelcited by 42
  6. 2023SALLM: Security Assessment of Generated Codecited by 35
  7. 2023Fully Autonomous Programming with Large Language Modelscited by 31
  8. 2023Are Emily and Greg Still More Employable than Lakisha and Jamal? Investigating Algorithmic Hiring Bias in the Era of ChatGPTcited by 23
  9. 2023Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generationcited by 22
  10. 2024Opening the AI black box: program synthesis via mechanistic interpretabilitycited by 19
  11. 2024Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projectscited by 18
  12. 2023Comparing Software Developers with ChatGPT: An Empirical Investigationcited by 11
12 more
  1. 2023Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Contentcited by 9
  2. 2023Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Modelscited by 8
  3. 2025Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Modelscited by 3
  4. 2023ChatGPT for GTFS: Benchmarking LLMs on GTFS Understanding and Retrievalcited by 1
  5. 2023Anticipating User Needs: Insights from Design Fiction on Conversational Agents for Computational Thinkingcited by 1
  6. 2024The Rise and Fall(?) of Software Engineeringcited by 1
  7. 2024Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasetscited by 1
  8. 2025The Role of GitHub Copilot on Software Development: A Perspective on Productivity, Security, Best Practices and Future Directionscited by 1
  9. 2023Towards Automatic Support of Software Model Evolution with Large Language~Modelscited by 0
  10. 2024On Languaging a Simulation Enginecited by 0
  11. 2024Optimizing Large Language Models for OpenAPI Code Completioncited by 0
  12. 2026Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Codecited by 0