mlsanityData to 14 Oct 2026

Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs

arXiv 2111.03922 ↗First posted 6 Nov 2021cs.SEcited by 25

Not on the map. Why?

Abstract (arXiv)

OpenAI's Codex, a GPT-3 like model trained on a large code corpus, has made headlines in and outside of academia. Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases. In this work, we want to investigate whether Codex is able to localize and fix bugs, a task of central interest in the field of automated program repair. Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java). We find that, despite not being trained for APR, Codex is surprisingly effective, and competitive with recent state of the art techniques. Our results also show that Codex is slightly more successful at repairing Python than Java.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Nov 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 7; 2025: 0; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  2. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  3. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
  4. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  5. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  6. 2023Large Language Models and Simple, Stupid Bugscited by 45
  7. 2023A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verificationcited by 42
  8. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  9. 2023Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyondcited by 20
  10. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  11. 2024LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Rewardcited by 16
  12. 2022Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluationcited by 14
13 more
  1. 2024Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Modelscited by 10
  2. 2023Extending the Frontier of ChatGPT: Code Generation and Debuggingcited by 9
  3. 2024Integrating Large Language Models with Internet of Things Applicationscited by 7
  4. 2022Follow-up Attention: An Empirical Study of Developer and Neural Model Code Explorationcited by 5
  5. 2024ChatGPT Incorrectness Detection in Software Reviewscited by 5
  6. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4
  7. 2023Chatbots As Fluent Polyglots: Revisiting Breakthrough Code Snippetscited by 3
  8. 2022JEMMA: An Extensible Java Dataset for ML4Code Applicationscited by 2
  9. 2024Benchmarking Educational Program Repaircited by 2
  10. 2026QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimizationcited by 2
  11. 2023Towards Automatically Addressing Self-Admitted Technical Debt: How Far Are We?cited by 1
  12. 2023Exploring Large Language Models in Resolving Environment-Related Crash Bugs: Localizing and Repairingcited by 1
  13. 2026Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repaircited by 0