# DABstep: Data Agent Benchmark for Multi-step Reasoning

arXiv: https://arxiv.org/abs/2506.23719 · HTML: https://mlsanity.com/paper/2506.23719/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal](https://mlsanity.com/paper/2406.14598/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)

## Built on by (most cited first, 30 of 46)

- [DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle](https://mlsanity.com/paper/2512.04324/) (2025)
- [AgenticData: An Agentic Data Analytics System for Heterogeneous Data](https://mlsanity.com/paper/2508.05002/) (2025)
- [DSGym: A Holistic Framework for Evaluating and Training Data Science Agents](https://mlsanity.com/paper/2601.16344/) (2026)
- [Automated Benchmark Auditing for AI Agents and Large Language Models](https://mlsanity.com/paper/2605.26079/) (2026)
- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheets](https://mlsanity.com/paper/2510.19247/) (2025)
- [DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysis](https://mlsanity.com/paper/2601.21403/) (2026)
- [LongDA: Benchmarking LLM Agents for Long-Document Data Analysis](https://mlsanity.com/paper/2601.02598/) (2026)
- [DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems](https://mlsanity.com/paper/2601.13591/) (2026)
- [Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models](https://mlsanity.com/paper/2602.02039/) (2026)
- [Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis](https://mlsanity.com/paper/2604.24198/) (2026)
- [DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science](https://mlsanity.com/paper/2602.24288/) (2026)
- [LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis](https://mlsanity.com/paper/2605.30434/) (2026)
- [Game-Theoretic Lens on LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2601.15047/) (2026)
- [PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?](https://mlsanity.com/paper/2605.08687/) (2026)
- [AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments](https://mlsanity.com/paper/2607.05174/) (2026)
- [Scaling Scientific Discovery Environments for Turn-Level Agentic RL](https://mlsanity.com/paper/2607.28990/) (2026)
- [DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces](https://mlsanity.com/paper/2608.03451/) (2026)
- [WebDS: An End-to-End Benchmark for Web-based Data Science](https://mlsanity.com/paper/2508.01222/) (2025)
- [ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding](https://mlsanity.com/paper/2510.13499/) (2025)
- [Unsupervised Skill Discovery for Agentic Data Analysis](https://mlsanity.com/paper/2606.06416/) (2026)
- [DSBC : Data Science task Benchmarking with Context engineering](https://mlsanity.com/paper/2507.23336/) (2025)
- [Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity](https://mlsanity.com/paper/2601.15728/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [VESTA: Visual Exploration with Statistical Tool Agents](https://mlsanity.com/paper/2606.00384/) (2026)
- [Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery](https://mlsanity.com/paper/2607.15766/) (2026)
- [DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?](https://mlsanity.com/paper/2608.10366/) (2026)
- [A Multi-Agent Approach to Neurological Clinical Reasoning](https://mlsanity.com/paper/2508.14063/) (2025)
- [COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding](https://mlsanity.com/paper/2603.21329/) (2026)
- [Structure-Grounded Knowledge Retrieval via Code Dependencies for Multi-Step Data Reasoning](https://mlsanity.com/paper/2604.10516/) (2026)

## Abstract (arXiv)

We introduce DABstep, a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks. DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation. Each task demands an iterative, multi-step problem-solving approach, testing capabilities in data manipulation, cross-referencing multiple sources, and precise result reporting. The benchmark provides a factoid-style answer format with automatic correctness checks for objective scoring at scale. We evaluate leading LLM-based agents, revealing a substantial performance gap: even the best agent achieves only 14.55% accuracy on the hardest tasks. We detail our benchmark's design, dataset composition, task formulation, evaluation protocol, report baseline results and analyze failure modes. DABstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.