mlsanityData to 7 Oct 2026

Is Self-Repair a Silver Bullet for Code Generation?

arXiv 2306.09896 ↗First posted 16 Jun 2023cs.CL cs.AI cs.PLcited by 191

Not on the map. Why?

Cited by 43 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Builds on 26 of its 70 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

26 of 70 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
14 more
  1. 2022CodeT: Code Generation with Generated Testscited by 369
  2. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  4. 2019SPoC: Search-based Pseudocode to Codecited by 176
  5. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  6. 2022Natural Language to Code Translation with Executioncited by 122
  7. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  8. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  9. 2022Coder Reviewer Reranking for Code Generationcited by 83
  10. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  11. 2019Getafix: Learning to Fix Bugs Automaticallycited by 40
  12. 2020Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesiscited by 39
  13. 2022Fault-Aware Neural Code Rankerscited by 33
  14. 2017Dynamic Neural Program Embedding for Program Repaircited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 63; 2025: 45; 2026: 71.

191 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  3. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  4. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  5. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  6. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  7. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  8. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  9. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  10. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  11. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  12. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
38 more
  1. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  2. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  3. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  4. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  5. 2024On Verbalized Confidence Scores for LLMscited by 83
  6. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  7. 2024AnalogCoder: Analog Circuit Design via Training-Free Code Generationcited by 82
  8. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  9. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  10. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  11. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  12. 2023Clover: Closed-Loop Verifiable Code Generationcited by 62
  13. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 50
  14. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  15. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 46
  16. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  17. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  18. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 38
  19. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 38
  20. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  21. 2024Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Provingcited by 35
  22. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 34
  23. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generationcited by 33
  24. 2024Automated Proof Generation for Rust Code via Self-Evolutioncited by 31
  25. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  26. 2024Is Programming by Example solved by LLMs?cited by 27
  27. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  28. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  29. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  30. 2024EVOR: Evolving Retrieval for Code Generationcited by 22
  31. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  32. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  33. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  34. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  35. 2024Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Waycited by 19
  36. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  37. 2025CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilationcited by 19
  38. 2023INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repaircited by 18

50 most cited of 191 All 191 (JSON)