mlsanityData to 7 Oct 2026

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

arXiv 2402.16906 ↗First posted 25 Feb 2024cs.SE cs.AI cs.CLcited by 123

Not on the map. Why?

Cited by 43 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

37 of 66 references.

References · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  12. 2023Teaching Large Language Models to Self-Debugcited by 963
25 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  2. 2023StarCoder: may the source be with you!cited by 790
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2022CodeT: Code Generation with Generated Testscited by 369
  6. 2020Unsupervised Translation of Programming Languagescited by 262
  7. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  8. 2022Binding Language Models in Symbolic Languagescited by 224
  9. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  10. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  11. 2022Natural Language to Code Translation with Executioncited by 122
  12. 2023InferFix: End-to-End Program Repair with LLMscited by 113
  13. 2022Coder Reviewer Reranking for Code Generationcited by 83
  14. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  15. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  16. 2017Code Completion with Neural Attention and Pointer Networkscited by 49
  17. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  18. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  19. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  20. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  21. 2024CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technologycited by 21
  22. 2023GrACE: Generation using Associated Code Editscited by 16
  23. 2023Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanationscited by 12
  24. 2022Automated Feedback Generation for Competition-Level Codecited by 7
  25. 2023A Survey of Prevent and Detect Access Control Vulnerabilitiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 62; 2026: 35.

123 papers.

Sort:

By area:

  1. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  3. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  4. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  5. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  6. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  7. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimizationcited by 84
  8. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 65
  9. 2025Assemble Your Crew: Automatic Multi-agent Communication Topology Design via Autoregressive Graph Generationcited by 57
  10. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  11. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  12. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
38 more
  1. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 45
  2. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  3. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
  4. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  5. 2025CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debuggingcited by 31
  6. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  7. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  8. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  9. 2024BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Taskscited by 19
  10. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  11. 2026In Line with Context: Repository-Level Code Generation via Context Inliningcited by 18
  12. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 17
  13. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 15
  14. 2025CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generationcited by 14
  15. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  16. 2025Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Teamcited by 13
  17. 2025VeriDebug: A Unified LLM for Verilog Debugging via Contrastive Embedding and Guided Correctioncited by 12
  18. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  19. 2025HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systemscited by 11
  20. 2026CodeTracer: Towards Traceable Agent Statescited by 11
  21. 2025LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctnesscited by 10
  22. 2025Enhancing LLM-Based Code Generation with Complexity Metrics: A Feedback-Driven Approachcited by 10
  23. 2025Execution Guided Line-by-Line Code Generationcited by 10
  24. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 9
  25. 2024Aligning CodeLLMs with Direct Preference Optimizationcited by 9
  26. 2024Defining and Evaluating Physical Safety for Large Language Modelscited by 9
  27. 2025NG-Router: Graph-Supervised Multi-Agent Collaboration for Nutrition Question Answeringcited by 9
  28. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generationcited by 8
  29. 2025DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automationcited by 8
  30. 2025Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latencycited by 8
  31. 2026AgentStepper: Interactive Debugging of Software Development Agentscited by 8
  32. 2024Automating Thought of Search: A Journey Towards Soundness and Completenesscited by 7
  33. 2025Successor-Generator Planning with LLM-generated Heuristicscited by 7
  34. 2025Large Language Model Guided Self-Debugging Code Generationcited by 7
  35. 2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
  36. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  37. 2024Experimenting with Multi-Agent Software Development: Towards a Unified Platformcited by 6
  38. 2024SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilotcited by 6

50 most cited of 123 All 123 (JSON)