# Automated Repair of Programs from Large Language Models

arXiv: https://arxiv.org/abs/2205.10583 · HTML: https://mlsanity.com/paper/2205.10583/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2022.

## Built on by (most cited first, 30 of 71)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [NExT: Teaching Large Language Models to Reason about Code Execution](https://mlsanity.com/paper/2404.14662/) (2024)
- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Code Repair with LLMs gives an Exploration-Exploitation Tradeoff](https://mlsanity.com/paper/2405.17503/) (2024)
- [Calibration and Correctness of Language Models for Code](https://mlsanity.com/paper/2402.02047/) (2024)
- [Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions](https://mlsanity.com/paper/2312.12450/) (2023)
- [A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification](https://mlsanity.com/paper/2305.14752/) (2023)
- [An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detection](https://mlsanity.com/paper/2406.06822/) (2024)
- [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023)
- [Fully Autonomous Programming with Large Language Models](https://mlsanity.com/paper/2304.10423/) (2023)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis](https://mlsanity.com/paper/2402.01386/) (2024)
- [Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps](https://mlsanity.com/paper/2309.04142/) (2023)
- [Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios](https://mlsanity.com/paper/2410.12468/) (2024)
- [ITER: Iterative Neural Repair for Multi-Location Patches](https://mlsanity.com/paper/2304.12015/) (2023)
- [Revisiting the Reliability of Psychological Scales on Large Language Models](https://mlsanity.com/paper/2305.19926/) (2023)
- [Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach](https://mlsanity.com/paper/2310.06680/) (2023)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions](https://mlsanity.com/paper/2304.03816/) (2023)
- [Aligning the Objective of LLM-based Program Repair](https://mlsanity.com/paper/2404.08877/) (2024)
- [Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment](https://mlsanity.com/paper/2502.07803/) (2025)
- [LLMs: A Game-Changer for Software Engineers?](https://mlsanity.com/paper/2411.00932/) (2024)
- [A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?](https://mlsanity.com/paper/2411.01414/) (2024)
- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)

## Abstract (arXiv)

Large language models such as Codex, have shown the capability to produce code for many programming tasks. However, the success rate of existing models is low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics, resulting in incorrect programs, or even programs which do not compile. In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance reliability in the code produced by large language models. Our study revealed that: (1) automatically generated code shares common programming mistakes with human-crafted solutions, indicating APR techniques may have potential to fix auto-generated code; (2) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports editing code, is similar to or better than existing Java repair tools TBar and Recoder in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions: (1) enhancing APR tools to surpass limitations in patch space (e.g., introducing more flexible fault localization) is desirable; (2) as large language models can derive more fix patterns by training on more data, future APR tools could shift focus from adding more fix patterns to synthesis/semantics based approaches, (3) combination of language models with APR to curate patch ingredients, is worth studying.