# Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs

arXiv: https://arxiv.org/abs/2409.10033 · HTML: https://mlsanity.com/paper/2409.10033/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [Automated Repair of Programs from Large Language Models](https://mlsanity.com/paper/2205.10583/) (2022)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [TBar: Revisiting Template-based Automated Program Repair](https://mlsanity.com/paper/1903.08409/) (2019)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [A Survey of Learning-based Automated Program Repair](https://mlsanity.com/paper/2301.03270/) (2023)
- [Conversational Automated Program Repair](https://mlsanity.com/paper/2301.13246/) (2023)
- [A Systematic Literature Review on Large Language Models for Automated Program Repair](https://mlsanity.com/paper/2405.01466/) (2024)
- [Neural Transfer Learning for Repairing Security Vulnerabilities in C Code](https://mlsanity.com/paper/2104.08308/) (2021)
- [Automatic Code Summarization via ChatGPT: How Far Are We?](https://mlsanity.com/paper/2305.12865/) (2023)
- [Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs](https://mlsanity.com/paper/1811.04211/) (2018)
- [A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair](https://mlsanity.com/paper/2310.08879/) (2023)
- [GAMMA: Revisiting Template-based Automated Program Repair via Mask Prediction](https://mlsanity.com/paper/2309.09308/) (2023)
- [AVATAR : Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations](https://mlsanity.com/paper/1812.07270/) (2018)
- [A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark](https://mlsanity.com/paper/1805.03454/) (2018)
- [CIRCLE: Continual Repair across Programming Languages](https://mlsanity.com/paper/2205.10956/) (2022)
- [On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs](https://mlsanity.com/paper/2008.00914/) (2020)
- [Pre-trained Model-based Automated Software Vulnerability Repair: How Far are We?](https://mlsanity.com/paper/2308.12533/) (2023)
- [You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems](https://mlsanity.com/paper/1812.07283/) (2018)
- [SeqTrans: Automatic Vulnerability Fix via Sequence to Sequence Learning](https://mlsanity.com/paper/2010.10805/) (2020)
- [TransRepair: Context-aware Program Repair for Compilation Errors](https://mlsanity.com/paper/2210.03986/) (2022)
- [No Man is an Island: Towards Fully Automatic Programming by Code Search, Code Generation and Program Repair](https://mlsanity.com/paper/2409.03267/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education](https://mlsanity.com/paper/2410.21287/) (2024)
- [Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark](https://mlsanity.com/paper/2512.05091/) (2025)
- [Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need](https://mlsanity.com/paper/2411.12448/) (2024)
- [SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair](https://mlsanity.com/paper/2602.23647/) (2026)
- [Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving](https://mlsanity.com/paper/2505.00031/) (2025)
- [Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index](https://mlsanity.com/paper/2503.05852/) (2025)
- [MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs](https://mlsanity.com/paper/2607.26591/) (2026)

## Abstract (arXiv)

LLMs have long demonstrated remarkable effectiveness in automatic program repair (APR), with OpenAI's ChatGPT being one of the most widely used models in this domain. Through continuous iterations and upgrades of GPT-family models, their performance in fixing bugs has already reached state-of-the-art levels. However, there are few works comparing the effectiveness and variations of different versions of GPT-family models on APR. In this work, inspired by the recent public release of the GPT-o1 models, we conduct the first study to compare the effectiveness of different versions of the GPT-family models in APR. We evaluate the performance of the latest version of the GPT-family models (i.e., O1-preview and O1-mini), GPT-4o, and the historical version of ChatGPT on APR. We conduct an empirical study of the four GPT-family models against other LLMs and APR techniques on the QuixBugs benchmark from multiple evaluation perspectives, including repair success rate, repair cost, response length, and behavior patterns. The results demonstrate that O1's repair capability exceeds that of prior GPT-family models, successfully fixing all 40 bugs in the benchmark. Our work can serve as a foundation for further in-depth exploration of the applications of GPT-family models in APR.