# DataSciBench: An LLM Agent Benchmark for Data Science

arXiv: https://arxiv.org/abs/2502.13897 · HTML: https://mlsanity.com/paper/2502.13897/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries](https://mlsanity.com/paper/2312.13671/) (2023)
- [NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts](https://mlsanity.com/paper/2405.04520/) (2024)

## Built on by (most cited first, 30 of 57)

- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [DeepAnalyze: Agentic Large Language Models for Autonomous Data Science](https://mlsanity.com/paper/2510.16872/) (2025)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games](https://mlsanity.com/paper/2506.03610/) (2025)
- [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025)
- [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025)
- [RExBench: Can coding agents autonomously implement AI research extensions?](https://mlsanity.com/paper/2506.22598/) (2025)
- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [DSGym: A Holistic Framework for Evaluating and Training Data Science Agents](https://mlsanity.com/paper/2601.16344/) (2026)
- [Scaling Generalist Data-Analytic Agents](https://mlsanity.com/paper/2509.25084/) (2025)
- [AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science](https://mlsanity.com/paper/2506.13992/) (2025)
- [Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study](https://mlsanity.com/paper/2506.19794/) (2025)
- [CLIMATEAGENT: Multi-Agent Orchestration for Complex Climate Data Science Workflows](https://mlsanity.com/paper/2511.20109/) (2025)
- [Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse](https://mlsanity.com/paper/2602.18710/) (2026)
- [DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytelling](https://mlsanity.com/paper/2604.06474/) (2026)
- [AgenticDataBench: A Comprehensive Benchmark for Data Agents](https://mlsanity.com/paper/2607.01647/) (2026)
- [LongDA: Benchmarking LLM Agents for Long-Document Data Analysis](https://mlsanity.com/paper/2601.02598/) (2026)
- [DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems](https://mlsanity.com/paper/2601.13591/) (2026)
- [Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models](https://mlsanity.com/paper/2602.02039/) (2026)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [Agentic AutoSurvey: Let LLMs Survey LLMs](https://mlsanity.com/paper/2509.18661/) (2025)
- [InnoGym: Benchmarking the Innovation Potential of AI Agents](https://mlsanity.com/paper/2512.01822/) (2025)
- [DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows](https://mlsanity.com/paper/2512.04416/) (2025)
- [DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science](https://mlsanity.com/paper/2602.24288/) (2026)
- [LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis](https://mlsanity.com/paper/2605.30434/) (2026)
- [PISA: A Pragmatic Psych-Inspired Unified Memory System for Enhanced AI Agency](https://mlsanity.com/paper/2510.15966/) (2025)
- [ML-Tool-Bench: Tool-Augmented Planning for ML Tasks](https://mlsanity.com/paper/2512.00672/) (2025)
- [How Well Do Multimodal Models Reason on ECG Signals?](https://mlsanity.com/paper/2603.00312/) (2026)
- [CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?](https://mlsanity.com/paper/2606.15300/) (2026)

## Abstract (arXiv)

This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Model (LLM) capabilities in data science. Recent related benchmarks have primarily focused on single tasks, easily obtainable ground truth, and straightforward evaluation metrics, which limits the scope of tasks that can be evaluated. In contrast, DataSciBench is constructed based on a more comprehensive and curated collection of natural and challenging prompts for uncertain ground truth and evaluation metrics. We develop a semi-automated pipeline for generating ground truth (GT) and validating evaluation metrics. This pipeline utilizes and implements an LLM-based self-consistency and human verification strategy to produce accurate GT by leveraging collected prompts, predefined task types, and aggregate functions (metrics). Furthermore, we propose an innovative Task - Function - Code (TFC) framework to assess each code execution outcome based on precisely defined metrics and programmatic rules. Our experimental framework involves testing 6 API-based models, 8 open-source general models, and 9 open-source code generation models using the diverse set of prompts we have gathered. This approach aims to provide a more comprehensive and rigorous evaluation of LLMs in data science, revealing their strengths and weaknesses. Experimental results demonstrate that API-based models outperform open-sourced models on all metrics and Deepseek-Coder-33B-Instruct achieves the highest score among open-sourced models. We release all code and data at https://github.com/THUDM/DataSciBench.