mlsanityData to 14 Oct 2026

Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors

arXiv 2503.22388 ↗First posted 28 Mar 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs' debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the future. DSDBench is publicly available at github.com/KevinCL16/DSDBench.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

18 of 30 references.

References · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  8. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  9. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
  10. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  11. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  12. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 102
6 more
  1. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  2. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 63
  3. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  4. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
  5. 2024PyBench: Evaluating LLM Agent on various real-world coding taskscited by 24
  6. 2024COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesiscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Discussion to Execution: Replicating Buggy and Correct Data Science Code ↗
  2. 2025Knowledge-Enhanced Program Repair for Data Science Code
  3. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
  4. 2026Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
  5. 2025AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science
  6. 2025DataSciBench: An LLM Agent Benchmark for Data Science
  7. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
  8. 2026Evaluating Agentic Code Repair Capabilities in Distributed Systems