# AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

arXiv: https://arxiv.org/abs/2603.19005 · HTML: https://mlsanity.com/paper/2603.19005/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions](https://mlsanity.com/paper/2410.20424/) (2024)
- [How do Data Science Workers Collaborate? Roles, Workflows, and Tools](https://mlsanity.com/paper/2001.06684/) (2020)
- [DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models](https://mlsanity.com/paper/2410.07331/) (2024)
- [DataSciBench: An LLM Agent Benchmark for Data Science](https://mlsanity.com/paper/2502.13897/) (2025)
- [Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making](https://mlsanity.com/paper/2208.07960/) (2022)
- [How Data Scientists Work Together With Domain Experts in Scientific Collaborations: To Find The Right Answer Or To Ask The Right Question?](https://mlsanity.com/paper/1909.03486/) (2019)
- [Evaluating Human-AI Collaboration: A Review and Methodological Framework](https://mlsanity.com/paper/2407.19098/) (2024)
- [SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning](https://mlsanity.com/paper/2410.17238/) (2024)
- [I-MCTS: Enhancing Agentic AutoML via Introspective Monte Carlo Tree Search](https://mlsanity.com/paper/2502.14693/) (2025)
- [SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science](https://mlsanity.com/paper/2503.23314/) (2025)
- [AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science](https://mlsanity.com/paper/2506.13992/) (2025)
- [HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI](https://mlsanity.com/paper/2501.15627/) (2025)
- [Can Agentic AI Match the Performance of Human Data Scientists?](https://mlsanity.com/paper/2512.20959/) (2025)

## Built on by (most cited first, 2 of 2)

- [LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis](https://mlsanity.com/paper/2605.30434/) (2026)
- [Nonuniformity Principle in Human-AI Coworking](https://mlsanity.com/paper/2607.16530/) (2026)

## Abstract (arXiv)

Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .