mlsanityData to 7 Oct 2026

Automated Repair of Programs from Large Language Models

arXiv 2205.10583 ↗First posted 21 May 2022cs.SEcited by 71

Not on the map. Why?

Abstract (arXiv)

Large language models such as Codex, have shown the capability to produce code for many programming tasks. However, the success rate of existing models is low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics, resulting in incorrect programs, or even programs which do not compile. In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance reliability in the code produced by large language models. Our study revealed that: (1) automatically generated code shares common programming mistakes with human-crafted solutions, indicating APR techniques may have potential to fix auto-generated code; (2) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports editing code, is similar to or better than existing Java repair tools TBar and Recoder in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions: (1) enhancing APR tools to surpass limitations in patch space (e.g., introducing more flexible fault localization) is desirable; (2) as large language models can derive more fix patterns by training on more data, future APR tools could shift focus from adding more fix patterns to synthesis/semantics based approaches, (3) combination of language models with APR to curate patch ingredients, is worth studying.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 22; 2025: 16; 2026: 14.

71 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  3. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 93
  4. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
  5. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  6. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  7. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  8. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  9. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 50
  10. 2024Calibration and Correctness of Language Models for Codecited by 49
  11. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 46
  12. 2023A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verificationcited by 42
38 more
  1. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  2. 2024An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detectioncited by 32
  3. 2023Fully Autonomous Programming with Large Language Modelscited by 31
  4. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  5. 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysiscited by 20
  6. 2023Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmapscited by 19
  7. 2023ITER: Iterative Neural Repair for Multi-Location Patchescited by 16
  8. 2023Revisiting the Reliability of Psychological Scales on Large Language Modelscited by 16
  9. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 16
  10. 2023Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approachcited by 15
  11. 2023Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptionscited by 14
  12. 2024CigaR: Cost-efficient Program Repair with LLMscited by 14
  13. 2024Are LLMs Correctly Integrated into Software Systems?cited by 13
  14. 2024Aligning the Objective of LLM-based Program Repaircited by 12
  15. 2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignmentcited by 12
  16. 2024LLMs: A Game-Changer for Software Engineers?cited by 11
  17. 2023Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Codecited by 9
  18. 2024RepairBench: Leaderboard of Frontier Models for Program Repaircited by 9
  19. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  20. 2025Investigating The Smells of LLM Generated Codecited by 9
  21. 2023Seeing Seeds Beyond Weeds: Green Teaming Generative AI for Beneficial Usescited by 8
  22. 2024Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugscited by 7
  23. 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
  24. 2025Agentic Services Computingcited by 6
  25. 2023Frustrated with Code Quality Issues? LLMs can Help!cited by 5
  26. 2023Configuration Validation with Large Language Modelscited by 4
  27. 2023LUNA: A Model-Based Universal Analysis Framework for Large Language Modelscited by 4
  28. 2025Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generationcited by 4
  29. 2025EditLord: Learning Code Transformation Rules for Code Editingcited by 4
  30. 2025PAGENT: Learning to Patch Software Engineering Agentscited by 4
  31. 2025Function-to-Style Guidance of LLMs for Code Translationcited by 4
  32. 2024Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translationcited by 3
  33. 2024CovRL: Fuzzing JavaScript Engines with Coverage-Guided Reinforcement Learning for LLM-based Mutationcited by 3
  34. 2024Automated Repair of AI Code with Large Language Models and Formal Verificationcited by 3
  35. 2025Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Modelscited by 3
  36. 2025Can Large Language Models Be Trusted as Evolutionary Optimizers for Network-Structured Combinatorial Problems?cited by 3
  37. 2024Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projectscited by 3
  38. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3

50 most cited of 71 All 71 (JSON)