mlsanityData to 7 Oct 2026

Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agents

arXiv 2506.08800 ↗First posted 10 Jun 2025cs.AI cs.CLcited by 4

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Data science aims to extract insights from data to support decision-making processes. Recently, Large Language Models (LLMs) have been increasingly used as assistants for data science, by suggesting ideas, techniques and small code snippets, or for the interpretation of results and reporting. Proper automation of some data-science activities is now promised by the rise of LLM agents, i.e., AI systems powered by an LLM equipped with additional affordances--such as code execution and knowledge bases--that can perform self-directed actions and interact with digital environments. In this paper, we survey the evaluation of LLM assistants and agents for data science. We find (1) a dominant focus on a small subset of goal-oriented activities, largely ignoring data management and exploratory activities; (2) a concentration on pure assistance or fully autonomous agents, without considering intermediate levels of human-AI collaboration; and (3) an emphasis on human substitution, therefore neglecting the possibility of higher levels of automation thanks to task transformation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 60 of its 75 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

60 of 75 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agents20 Feb
  2. 2025CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories10 Feb
  3. 2024A Survey on Large Language Model-based Agents for Statistics and Data Science18 Dec
  4. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts22 Nov
  5. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models9 Oct
  6. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct
  7. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery7 Oct
  8. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?12 Sep
  9. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories11 Sep
  10. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science19 Aug
  11. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation8 Jul
  12. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Models1 Jul
4 more
  1. 2024Benchmarking Data Science Agents27 Feb
  2. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks10 Jan
  3. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  12. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
32 more
  1. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  2. 2025Survey on Evaluation of LLM-based Agentscited by 190
  3. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 182
  4. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  5. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  6. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  7. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
  8. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  9. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  10. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 89
  11. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  12. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  13. 2023LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Modelscited by 80
  14. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  15. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  16. 2023Is GPT-4 a Good Data Analyst?cited by 69
  17. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  18. 2022AMLB: an AutoML Benchmarkcited by 55
  19. 2022The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligencecited by 42
  20. 2022Training and Evaluating a Jupyter Notebook Data Science Assistantcited by 31
  21. 2021nvBench: A Large-Scale Synthesized Dataset for Cross-Domain Natural Language to Visualization Taskcited by 27
  22. 2021Automating Data Science: Prospects and Challengescited by 17
  23. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
  24. 2025DSCodeBench: A Realistic Benchmark for Data Science Code Generationcited by 15
  25. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysiscited by 11
  26. 2025Measuring AI agent autonomy: Towards a scalable approach with code inspectioncited by 10
  27. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 10
  28. 2025StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysiscited by 8
  29. 2025AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Sciencecited by 7
  30. 2025HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AIcited by 5
  31. 2025BIASINSPECTOR: Detecting Bias in Structured Data through LLM Agentscited by 4
  32. 2024Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientistscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Modelscited by 4
  2. 2025ORCA: ORchestrating Causal Agentcited by 1
  3. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  4. 2026Skill-based Agentic Evaluation for Real-time Data Science Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions
  2. 2025Large Language Model-based Data Science Agent: A Survey
  3. 2025AutoMind: Adaptive Knowledgeable Agent for Automated Data Science
  4. 2025DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automation
  5. 2026MUSE: An Interactive Meta-Agent for Understanding and Steering LLM-powered Data Science Systems
  6. 2026DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation
  7. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning
  8. 2024Data Interpreter: An LLM Agent For Data Science