# Repairing Bugs in Python Assignments Using Large Language Models

arXiv: https://arxiv.org/abs/2209.14876 · HTML: https://mlsanity.com/paper/2209.14876/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [Dynamic Neural Program Embedding for Program Repair](https://mlsanity.com/paper/1711.07163/) (2017)
- [Automated Clustering and Program Repair for Introductory Programming Assignments](https://mlsanity.com/paper/1603.03165/) (2016)
- [Learning Syntactic Program Transformations from Examples](https://mlsanity.com/paper/1608.09000/) (2016)
- [Multi-modal Program Inference: a Marriage of Pre-trainedLanguage Models and Component-based Synthesis](https://mlsanity.com/paper/2109.02445/) (2021)
- [sk_p: a neural program corrector for MOOCs](https://mlsanity.com/paper/1607.02902/) (2016)
- [Data-Driven Feedback Generation for Introductory Programming Exercises](https://mlsanity.com/paper/1711.07148/) (2017)
- [Verifix: Verified Repair of Programming Assignments](https://mlsanity.com/paper/2106.16199/) (2021)
- [Automated Feedback Generation for Competition-Level Code](https://mlsanity.com/paper/2206.01848/) (2022)
- [FAPR: Fast and Accurate Program Repair for Introductory Programming Courses](https://mlsanity.com/paper/2107.06550/) (2021)
- [MACER: A Modular Framework for Accelerated Compilation Error Repair](https://mlsanity.com/paper/2005.14015/) (2020)

## Built on by (most cited first, 28 of 28)

- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [RLTF: Reinforcement Learning from Unit Test Feedback](https://mlsanity.com/paper/2307.04349/) (2023)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)
- [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Refactoring Programs Using Large Language Models with Few-Shot Examples](https://mlsanity.com/paper/2311.11690/) (2023)
- [Performance-Aligned LLMs for Generating Fast Code](https://mlsanity.com/paper/2404.18864/) (2024)
- [Evaluating ChatGPT and GPT-4 for Visual Programming](https://mlsanity.com/paper/2308.02522/) (2023)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [A review on the use of large language models as virtual tutors](https://mlsanity.com/paper/2405.11983/) (2024)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments](https://mlsanity.com/paper/2308.08572/) (2023)
- [Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming](https://mlsanity.com/paper/2310.10690/) (2023)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generation](https://mlsanity.com/paper/2312.05356/) (2023)
- [Toward Reproducing Network Research Results Using Large Language Models](https://mlsanity.com/paper/2309.04716/) (2023)
- [Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs](https://mlsanity.com/paper/2405.13932/) (2024)
- [Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education](https://mlsanity.com/paper/2505.18220/) (2025)
- [PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritization](https://mlsanity.com/paper/2505.05584/) (2025)
- [Peer-aided Repairer: Empowering Large Language Models to Repair Advanced Student Assignments](https://mlsanity.com/paper/2404.01754/) (2024)
- [Bug-Report-Driven Fault Localization: Industrial Benchmarking and Lesson Learned at ABB Robotics](https://mlsanity.com/paper/2604.25700/) (2026)
- [An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration](https://mlsanity.com/paper/2606.12425/) (2026)

## Abstract (arXiv)

Students often make mistakes on their introductory programming assignments as part of their learning process. Unfortunately, providing custom repairs for these mistakes can require a substantial amount of time and effort from class instructors. Automated program repair (APR) techniques can be used to synthesize such fixes. Prior work has explored the use of symbolic and neural techniques for APR in the education domain. Both types of approaches require either substantial engineering efforts or large amounts of data and training. We propose to use a large language model trained on code, such as Codex, to build an APR system -- MMAPR -- for introductory Python programming assignments. Our system can fix both syntactic and semantic mistakes by combining multi-modal prompts, iterative querying, test-case-based selection of few-shots, and program chunking. We evaluate MMAPR on 286 real student programs and compare to a baseline built by combining a state-of-the-art Python syntax repair engine, BIFI, and state-of-the-art Python semantic repair engine for student assignments, Refactory. We find that MMAPR can fix more programs and produce smaller patches on average.