mlsanityData to 7 Oct 2026

Benchmarking Data Science Agents

arXiv 2402.17168 ↗First posted 27 Feb 2024cs.AI cs.CLcited by 30

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

In the era of data-driven decision-making, the complexity of data analysis necessitates advanced expertise and tools of data science, presenting significant challenges even for specialists. Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing. Yet their practical efficacy remains constrained by the varied demands of real-world applications and complicated analytical process. In this paper, we introduce DSEval -- a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents throughout the entire data science lifecycle. Incorporating a novel bootstrapped annotation method, we streamline dataset preparation, improve the evaluation coverage, and expand benchmarking comprehensiveness. Our findings uncover prevalent obstacles and provide critical insights to inform future advancements in the field.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 33 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

18 of 33 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  12. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
6 more
  1. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  2. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  3. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  4. 2023Is GPT-4 a Good Data Analyst?cited by 69
  5. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  6. 2023Experiential Co-Learning of Software-Developing Agentscited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 18; 2026: 7.

30 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  3. 2024VisEval: A Benchmark for Data Visualization in the Era of Large Language Modelscited by 43
  4. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 38
  5. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  6. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generationcited by 30
  7. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  8. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  9. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  10. 2025Exploring LLM Agents for Cleaning Tabular Machine Learning Datasetscited by 20
  11. 2025LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Sciencecited by 15
  12. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
18 more
  1. 2024DataLab: A Unified Platform for LLM-Powered Business Intelligencecited by 7
  2. 2026Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiversecited by 7
  3. 2026DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytellingcited by 7
  4. 2025AutoG: Towards automatic graph construction from tabular datacited by 6
  5. 2025DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queriescited by 6
  6. 2026DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysiscited by 6
  7. 2024TrajAgent: An LLM-Agent Framework for Trajectory Modeling via Large-and-Small Model Collaborationcited by 5
  8. 2024TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Modelscited by 5
  9. 2025DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulationcited by 4
  10. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  11. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  12. 2026DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Sciencecited by 3
  13. 2026PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?cited by 3
  14. 2026Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuelcited by 1
  15. 2025CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generationcited by 0
  16. 2025Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysiscited by 0
  17. 2025InData: Towards Secure Multi-Step, Tool-Based Data Analysiscited by 0
  18. 2026DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DSBC : Data Science task Benchmarking with Context engineering
  2. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
  3. 2026AgenticDataBench: A Comprehensive Benchmark for Data Agents
  4. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
  5. 2025DataSciBench: An LLM Agent Benchmark for Data Science
  6. 2024A Survey on Large Language Model-based Agents for Statistics and Data Science
  7. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models
  8. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions