mlsanityData to 14 Oct 2026

DebugBench: Evaluating Debugging Capability of Large Language Models

arXiv 2401.04621 ↗First posted 9 Jan 2024cs.SE cs.AI cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce `DebugBench', an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and four open-source models in a zero-shot scenario. We find that (1) while closed-source models exhibit inferior debugging performance compared to humans, open-source models relatively lower pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 13 of its 35 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

13 of 35 references.

References · 13

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2023Teaching Large Language Models to Self-Debugcited by 988
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  8. 2023StarCoder: may the source be with you!cited by 799
  9. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  10. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  11. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 110
  12. 2019TBar: Revisiting Template-based Automated Program Repaircited by 65
1 more
  1. 2023A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repaircited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 35; 2026: 23.

76 papers.

Sort:

By area:

  1. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  2. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 85
  3. 2025LifelongAgentBench: Evaluating LLM Agents as Lifelong Learnerscited by 78
  4. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  5. 2024AutoVerus: Automated Proof Generation for Rust Codecited by 49
  6. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  7. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  8. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  9. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  10. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 20
  11. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 20
  12. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
38 more
  1. 2025RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solvingcited by 19
  2. 2024VersiCode: Towards Version-controllable Code Generationcited by 17
  3. 2025Learning to Generate Unit Tests for Automated Debuggingcited by 17
  4. 2025SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applicationscited by 16
  5. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  6. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  8. 2025TOM-SWE: User Mental Modeling For Software Engineering Agentscited by 13
  9. 2024Aligning the Objective of LLM-based Program Repaircited by 12
  10. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  11. 2024Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Modelscited by 10
  12. 2024R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Modelscited by 9
  13. 2024Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Codecited by 9
  14. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 9
  15. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  16. 2025Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Modelscited by 7
  17. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Callingcited by 7
  18. 2026The Value Axis: Language Models Encode Whether They're on the Right Trackcited by 7
  19. 2024LLeMpower: Understanding Disparities in the Control and Access of Large Language Modelscited by 5
  20. 2025IFEvalCode: Controlled Code Generationcited by 5
  21. 2026Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMscited by 5
  22. 2026PACE: A Proxy for Agentic Capability Evaluationcited by 5
  23. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  24. 2025Improving Deep Assertion Generation via Fine-Tuning Retrieval-Augmented Pre-trained Language Modelscited by 4
  25. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  26. 2025Assessing the Impact of Code Changes on the Fault Localizability of Large Language Modelscited by 4
  27. 2025An LLM-enabled Multi-Agent Autonomous Mechatronics Design Frameworkcited by 4
  28. 2025NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debuggingcited by 4
  29. 2025The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Studycited by 4
  30. 2025Cognitive Agents Powered by Large Language Models for Agile Software Project Managementcited by 4
  31. 2025OntoLogX: Ontology-Guided Knowledge Graph Extraction from Cybersecurity Logs with Large Language Modelscited by 4
  32. 2026Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Modelscited by 4
  33. 2024Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programmingcited by 3
  34. 2024Multi-Programming Language Ensemble for Code Generation in Large Language Modelcited by 3
  35. 2025MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarioscited by 3
  36. 2025Multilingual Multimodal Software Developer for Code Generationcited by 3
  37. 2025Improving Retrieval-Augmented Deep Assertion Generation via Joint Trainingcited by 2
  38. 2025RefactorCoderQA: Benchmarking LLMs for Multi-Domain Coding Question Solutions in Cloud and Edge Deploymentcited by 2

50 most cited of 76 All 76 (JSON)