# Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming

arXiv: https://arxiv.org/abs/2111.07875 · HTML: https://mlsanity.com/paper/2111.07875/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [PyMT5: multi-mode translation of natural language and Python code with transformers](https://mlsanity.com/paper/2010.03150/) (2020)
- [SourceFinder: Finding Malware Source-Code from Publicly Available Repositories](https://mlsanity.com/paper/2005.14311/) (2020)
- [PSB2: The Second Program Synthesis Benchmark Suite](https://mlsanity.com/paper/2106.06086/) (2021)
- [Problem-solving benefits of down-sampled lexicase selection](https://mlsanity.com/paper/2106.06085/) (2021)
- [TensorGP -- Genetic Programming Engine in TensorFlow](https://mlsanity.com/paper/2103.07512/) (2021)
- [Recent Developments in Program Synthesis with Evolutionary Algorithms](https://mlsanity.com/paper/2108.12227/) (2021)

## Built on by (most cited first, 24 of 24)

- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop](https://mlsanity.com/paper/2311.16822/) (2023)
- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [Evolving Code with A Large Language Model](https://mlsanity.com/paper/2401.07102/) (2024)
- [SALLM: Security Assessment of Generated Code](https://mlsanity.com/paper/2311.00889/) (2023)
- [Fully Autonomous Programming with Large Language Models](https://mlsanity.com/paper/2304.10423/) (2023)
- [Are Emily and Greg Still More Employable than Lakisha and Jamal? Investigating Algorithmic Hiring Bias in the Era of ChatGPT](https://mlsanity.com/paper/2310.05135/) (2023)
- [Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation](https://mlsanity.com/paper/2306.15121/) (2023)
- [Opening the AI black box: program synthesis via mechanistic interpretability](https://mlsanity.com/paper/2402.05110/) (2024)
- [Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects](https://mlsanity.com/paper/2406.17910/) (2024)
- [Comparing Software Developers with ChatGPT: An Empirical Investigation](https://mlsanity.com/paper/2305.11837/) (2023)
- [Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Content](https://mlsanity.com/paper/2309.02524/) (2023)
- [Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Models](https://mlsanity.com/paper/2301.13382/) (2023)
- [Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models](https://mlsanity.com/paper/2503.07693/) (2025)
- [ChatGPT for GTFS: Benchmarking LLMs on GTFS Understanding and Retrieval](https://mlsanity.com/paper/2308.02618/) (2023)
- [Anticipating User Needs: Insights from Design Fiction on Conversational Agents for Computational Thinking](https://mlsanity.com/paper/2311.06887/) (2023)
- [The Rise and Fall(?) of Software Engineering](https://mlsanity.com/paper/2406.10141/) (2024)
- [Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets](https://mlsanity.com/paper/2406.17651/) (2024)
- [The Role of GitHub Copilot on Software Development: A Perspective on Productivity, Security, Best Practices and Future Directions](https://mlsanity.com/paper/2502.13199/) (2025)
- [Towards Automatic Support of Software Model Evolution with Large Language~Models](https://mlsanity.com/paper/2312.12404/) (2023)
- [On Languaging a Simulation Engine](https://mlsanity.com/paper/2402.16482/) (2024)
- [Optimizing Large Language Models for OpenAPI Code Completion](https://mlsanity.com/paper/2405.15729/) (2024)
- [Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Code](https://mlsanity.com/paper/2601.19264/) (2026)

## Abstract (arXiv)

GitHub Copilot, an extension for the Visual Studio Code development environment powered by the large-scale language model Codex, makes automatic program synthesis available for software developers. This model has been extensively studied in the field of deep learning, however, a comparison to genetic programming, which is also known for its performance in automatic program synthesis, has not yet been carried out. In this paper, we evaluate GitHub Copilot on standard program synthesis benchmark problems and compare the achieved results with those from the genetic programming literature. In addition, we discuss the performance of both approaches. We find that the performance of the two approaches on the benchmark problems is quite similar, however, in comparison to GitHub Copilot, the program synthesis approaches based on genetic programming are not yet mature enough to support programmers in practical software development. Genetic programming usually needs a huge amount of expensive hand-labeled training cases and takes too much time to generate solutions. Furthermore, source code generated by genetic programming approaches is often bloated and difficult to understand. For future work on program synthesis with genetic programming, we suggest researchers to focus on improving the execution time, readability, and usability.