mlsanityData to 7 Oct 2026

Teaching Large Language Models to Self-Debug

arXiv 2304.05128 ↗First posted 11 Apr 2023cs.CL cs.AIcited by 963

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.

Key facts. Cited by 963 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 63 of its 75 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

63 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LEVER: Learning to Verify Language-to-Code Generation with Execution16 Feb
  2. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2022PAL: Program-aided Language Modelscited by 635
  9. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  10. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  11. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  12. 2022Self-critiquing models for assisting human evaluatorscited by 372
  13. 2023Language Models can Solve Computer Taskscited by 371
  14. 2022CodeT: Code Generation with Generated Testscited by 369
  15. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  16. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  17. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  18. 2022Generating Sequences by Learning to Self-Correctcited by 271
  19. 2020Unsupervised Translation of Programming Languagescited by 262
  20. 2023Pretraining Language Models with Human Preferencescited by 245
  21. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  22. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  23. 2016Language to Logical Form with Neural Attentioncited by 158
  24. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  25. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  26. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  27. 2022Natural Language to Code Translation with Executioncited by 122
  28. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  29. 2023Learning Performance-Improving Code Editscited by 103
  30. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  31. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  32. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  33. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  34. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  35. 2022A data-driven approach for learning to control computerscited by 87
  36. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  37. 2022Coder Reviewer Reranking for Code Generationcited by 83
  38. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 144; 2024: 286; 2025: 244; 2026: 289.

963 papers.

Sort:

By area:

  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  8. 2023Large Language Models as Optimizerscited by 830
  9. 2024A Survey on Large Language Models for Code Generationcited by 646
  10. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  11. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
38 more
  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2023Cognitive Architectures for Language Agentscited by 422
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  4. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  5. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  6. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  7. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  8. 2023Large Language Models as Tool Makerscited by 293
  9. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  10. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  11. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  12. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  13. 2023Challenges and Applications of Large Language Modelscited by 251
  14. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  15. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  16. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  17. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  18. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  19. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  20. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  21. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  22. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  23. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  24. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  25. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  26. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  27. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  28. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  29. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  30. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  31. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  32. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  33. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  34. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  35. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  36. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  37. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  38. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140

50 most cited of 963 All 963 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisit Self-Debugging with Self-Generated Tests for Code Generation
  2. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation
  3. 2023Self-planning Code Generation with Large Language Models
  4. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  5. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
  6. 2024CodeMind: Evaluating Large Language Models for Code Reasoning
  7. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  8. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancement