mlsanityData to 7 Oct 2026

DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models

arXiv 2410.07331 ↗First posted 9 Oct 2024cs.CL cs.AIcited by 57

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

We introduce DA-Code, a code generation benchmark specifically designed to assess LLMs on agent-based data science tasks. This benchmark features three core elements: First, the tasks within DA-Code are inherently challenging, setting them apart from traditional code generation tasks and demanding advanced coding skills in grounding and planning. Second, examples in DA-Code are all based on real and diverse data, covering a wide range of complex data wrangling and analytics tasks. Third, to solve the tasks, the models must utilize complex data science programming languages, to perform intricate data processing and derive the answers. We set up the benchmark in a controllable and executable environment that aligns with real-world data analysis scenarios and is scalable. The annotators meticulously design the evaluation suite to ensure the accuracy and robustness of the evaluation. We develop the DA-Agent baseline. Experiments show that although the baseline performs better than other existing frameworks, using the current best LLMs achieves only 30.5% accuracy, leaving ample room for improvement. We release our benchmark at https://da-code-bench.github.io.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 32 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

20 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?15 Jul
  2. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks10 Jan

Other references · 18

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  10. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  11. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  12. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
6 more
  1. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  2. 2023Planning with Large Language Models for Code Generationcited by 179
  3. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  4. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  5. 2021AutoDS: Towards Human-Centered Automation of Data Sciencecited by 32
  6. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 29; 2026: 27.

57 papers.

Sort:

By area:

  1. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  4. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  5. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 52
  6. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  7. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  8. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  9. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27
  10. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  11. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 22
  12. 2026ReCreate: Reasoning and Creating Domain Agents Driven by Experiencecited by 22
38 more
  1. 2025Exploring LLM Agents for Cleaning Tabular Machine Learning Datasetscited by 20
  2. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 18
  3. 2025ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelinescited by 15
  4. 2025LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Sciencecited by 15
  5. 2025CoDA: Agentic Systems for Collaborative Data Visualizationcited by 14
  6. 2026DSGym: A Holistic Framework for Evaluating and Training Data Science Agentscited by 14
  7. 2025Byzantine-Robust Decentralized Coordination of LLM Agentscited by 13
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  9. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  10. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11
  11. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  12. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 9
  13. 2026Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiversecited by 7
  14. 2025AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discoverycited by 6
  15. 2025An Evaluation-Centric Paradigm for Scientific Visualization Agentscited by 6
  16. 2025DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queriescited by 6
  17. 2026TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Codecited by 6
  18. 2025CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournamentscited by 5
  19. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  20. 2026PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generationcited by 4
  21. 2025ConDABench: Interactive Evaluation of Language Models for Data Analysiscited by 3
  22. 2025Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysiscited by 3
  23. 2026DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Sciencecited by 3
  24. 2026CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?cited by 3
  25. 2026AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Sciencecited by 2
  26. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 2
  27. 2026Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agentscited by 2
  28. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environmentscited by 2
  29. 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 2
  30. 2026Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuelcited by 1
  31. 2026SciNav: A General Agent Framework for Scientific Coding Taskscited by 1
  32. 2026Auto-Configuring Scientific Simulators with Lightweight Coding-Agent Adapterscited by 1
  33. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  34. 2025CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generationcited by 0
  35. 2025WebDS: An End-to-End Benchmark for Web-based Data Sciencecited by 0
  36. 2025ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generationcited by 0
  37. 2025SoK: Understanding (New) Security Issues Across AI4Code Use Casescited by 0
  38. 2026Autonomous Data Processing using Meta-Agentscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FormulaCode: Evaluating Agentic Optimization on Large Codebases
  2. 2025DataSciBench: An LLM Agent Benchmark for Data Science
  3. 2026AgenticDataBench: A Comprehensive Benchmark for Data Agents
  4. 2024Data Interpreter: An LLM Agent For Data Science
  5. 2024Benchmarking Data Science Agents
  6. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
  7. 2025DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
  8. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions