mlsanityData to 7 Oct 2026

DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?

arXiv 2409.07703 ↗First posted 12 Sep 2024cs.AI cs.CLcited by 119

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the recent trend of building agents for targeted applications such as shopping assistants or AI software engineers. Recently, many data science benchmarks have been proposed to investigate their performance in the data science domain. However, existing data science benchmarks still fall short when compared to real-world data science applications due to their simplified settings. To bridge this gap, we introduce DSBench, a comprehensive benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. DSBench offers a realistic setting by encompassing long contexts, multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG). These findings underscore the need for further advancements in developing more practical, intelligent, and autonomous data science agents.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 2 of its 71 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

2 of 71 references.

References · 2

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 52; 2026: 63.

119 papers.

Sort:

By area:

  1. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  2. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  3. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  4. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  5. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  6. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  7. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  8. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  9. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  10. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  11. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  12. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 52
38 more
  1. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 49
  2. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  3. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 38
  4. 2025R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Sciencecited by 33
  5. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  6. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  7. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  8. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27
  9. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  10. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  11. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  12. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 19
  13. 2024AIGS: Generating Science from AI-Powered Automated Falsificationcited by 17
  14. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  15. 2025How Far Are AI Scientists from Changing the World?cited by 15
  16. 2025LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Sciencecited by 15
  17. 2025TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agentscited by 14
  18. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  19. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11
  20. 2025CellForge: Agentic Design of Virtual Cell Modelscited by 11
  21. 2025Scaling Generalist Data-Analytic Agentscited by 11
  22. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  23. 2025Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystemscited by 9
  24. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  25. 2025MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipelinecited by 9
  26. 2025A Survey of AI Scientistscited by 9
  27. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 9
  28. 2025BioDSA-1K: Benchmarking Data Science Agents for Biomedical Researchcited by 7
  29. 2025AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Sciencecited by 7
  30. 2025SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Modelscited by 7
  31. 2025KompeteAI: Accelerated Autonomous Multi-Agent System for End-to-End Pipeline Generation for Machine Learning Problemscited by 7
  32. 2026Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiversecited by 7
  33. 2026Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineeringcited by 7
  34. 2025When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Taskscited by 6
  35. 2026LongDA: Benchmarking LLM Agents for Long-Document Data Analysiscited by 6
  36. 2026FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agentscited by 6
  37. 2026FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamicscited by 6
  38. 2026DeployBench: Benchmarking LLM Agents for Research Artifact Deploymentcited by 6

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Data Science Agents
  2. 2025DataSciBench: An LLM Agent Benchmark for Data Science
  3. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning
  4. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions
  5. 2025DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries
  6. 2024A Survey on Large Language Model-based Agents for Statistics and Data Science
  7. 2026DSGym: A Holistic Framework for Evaluating and Training Data Science Agents
  8. 2023AgentBench: Evaluating LLMs as Agents