mlsanityData to 7 Oct 2026

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

arXiv 2603.19005 ↗First posted 19 Mar 2026cs.LG cs.AI stat.MEcited by 2

Not on the map. Why?

Abstract (arXiv)

Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

23 of 55 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  5. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  6. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
  7. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  8. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  9. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 119
  10. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
  11. 2024AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitionscited by 70
  12. 2020How do Data Science Workers Collaborate? Roles, Workflows, and Toolscited by 64
11 more
  1. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 57
  2. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 53
  3. 2022Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Makingcited by 33
  4. 2019How Data Scientists Work Together With Domain Experts in Scientific Collaborations: To Find The Right Answer Or To Ask The Right Question?cited by 30
  5. 2024Evaluating Human-AI Collaboration: A Review and Methodological Frameworkcited by 25
  6. 2024SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learningcited by 23
  7. 2025I-MCTS: Enhancing Agentic AutoML via Introspective Monte Carlo Tree Searchcited by 14
  8. 2025SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Sciencecited by 7
  9. 2025AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Sciencecited by 7
  10. 2025HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AIcited by 5
  11. 2025Can Agentic AI Match the Performance of Human Data Scientists?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysiscited by 4
  2. 2026Nonuniformity Principle in Human-AI Coworkingcited by 0