# DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2410.07331 · HTML: https://mlsanity.com/paper/2410.07331/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 20 of 20)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?](https://mlsanity.com/paper/2407.10956/) (2024)
- [AutoDS: Towards Human-Centered Automation of Data Science](https://mlsanity.com/paper/2101.05273/) (2021)
- [Competition-Level Problems are Effective LLM Evaluators](https://mlsanity.com/paper/2312.02143/) (2023)

## Built on by (most cited first, 30 of 58)

- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement](https://mlsanity.com/paper/2506.15692/) (2025)
- [DABstep: Data Agent Benchmark for Multi-step Reasoning](https://mlsanity.com/paper/2506.23719/) (2025)
- [AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents](https://mlsanity.com/paper/2602.06855/) (2026)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?](https://mlsanity.com/paper/2504.09702/) (2025)
- [ReCreate: Reasoning and Creating Domain Agents Driven by Experience](https://mlsanity.com/paper/2601.11100/) (2026)
- [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025)
- [DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle](https://mlsanity.com/paper/2512.04324/) (2025)
- [Exploring LLM Agents for Cleaning Tabular Machine Learning Datasets](https://mlsanity.com/paper/2503.06664/) (2025)
- [RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation](https://mlsanity.com/paper/2509.16198/) (2025)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines](https://mlsanity.com/paper/2504.04808/) (2025)
- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [CoDA: Agentic Systems for Collaborative Data Visualization](https://mlsanity.com/paper/2510.03194/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [DSGym: A Holistic Framework for Evaluating and Training Data Science Agents](https://mlsanity.com/paper/2601.16344/) (2026)
- [Byzantine-Robust Decentralized Coordination of LLM Agents](https://mlsanity.com/paper/2507.14928/) (2025)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists](https://mlsanity.com/paper/2506.08140/) (2025)
- [SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents](https://mlsanity.com/paper/2603.29139/) (2026)
- [PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation](https://mlsanity.com/paper/2605.09636/) (2026)
- [Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse](https://mlsanity.com/paper/2602.18710/) (2026)
- [AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery](https://mlsanity.com/paper/2504.07421/) (2025)
- [An Evaluation-Centric Paradigm for Scientific Visualization Agents](https://mlsanity.com/paper/2509.15160/) (2025)
- [DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries](https://mlsanity.com/paper/2509.21825/) (2025)
- [TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code](https://mlsanity.com/paper/2602.06875/) (2026)

## Abstract (arXiv)

We introduce DA-Code, a code generation benchmark specifically designed to assess LLMs on agent-based data science tasks. This benchmark features three core elements: First, the tasks within DA-Code are inherently challenging, setting them apart from traditional code generation tasks and demanding advanced coding skills in grounding and planning. Second, examples in DA-Code are all based on real and diverse data, covering a wide range of complex data wrangling and analytics tasks. Third, to solve the tasks, the models must utilize complex data science programming languages, to perform intricate data processing and derive the answers. We set up the benchmark in a controllable and executable environment that aligns with real-world data analysis scenarios and is scalable. The annotators meticulously design the evaluation suite to ensure the accuracy and robustness of the evaluation. We develop the DA-Agent baseline. Experiments show that although the baseline performs better than other existing frameworks, using the current best LLMs achieves only 30.5% accuracy, leaving ample room for improvement. We release our benchmark at https://da-code-bench.github.io.