mlsanityData to 14 Oct 2026

DABstep: Data Agent Benchmark for Multi-step Reasoning

arXiv 2506.23719 ↗First posted 30 Jun 2025cs.LG cs.AIcited by 46

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

We introduce DABstep, a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks. DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation. Each task demands an iterative, multi-step problem-solving approach, testing capabilities in data manipulation, cross-referencing multiple sources, and precise result reporting. The benchmark provides a factoid-style answer format with automatic correctness checks for objective scoring at scale. We evaluate leading LLM-based agents, revealing a substantial performance gap: even the best agent achieves only 14.55% accuracy on the hardest tasks. We detail our benchmark's design, dataset composition, task formulation, evaluation protocol, report baseline results and analyze failure modes. DABstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 38 of its 49 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

38 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?12 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
25 more
  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  4. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  5. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  7. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  8. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  9. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  10. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  11. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  12. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  13. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  14. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  15. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  16. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  17. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  18. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  19. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 77
  20. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 58
  21. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 56
  22. 2019On the Difficulty of Evaluating Baselines: A Study on Recommender Systemscited by 46
  23. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  24. 2024Benchmarking Data Science Agentscited by 30
  25. 2024WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Taskscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 38.

46 papers.

Sort:

By area:

  1. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 23
  2. 2025AgenticData: An Agentic Data Analytics System for Heterogeneous Datacited by 22
  3. 2026DSGym: A Holistic Framework for Evaluating and Training Data Science Agentscited by 15
  4. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 12
  5. 2025Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflowscited by 10
  6. 2025SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheetscited by 8
  7. 2026DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysiscited by 7
  8. 2026LongDA: Benchmarking LLM Agents for Long-Document Data Analysiscited by 6
  9. 2026DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problemscited by 6
  10. 2026Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Modelscited by 5
  11. 2026Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysiscited by 5
  12. 2026DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Sciencecited by 4
34 more
  1. 2026LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysiscited by 4
  2. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systemscited by 3
  3. 2026PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?cited by 3
  4. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environmentscited by 3
  5. 2026Scaling Scientific Discovery Environments for Turn-Level Agentic RLcited by 3
  6. 2026DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspacescited by 3
  7. 2025WebDS: An End-to-End Benchmark for Web-based Data Sciencecited by 2
  8. 2025ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understandingcited by 2
  9. 2026Unsupervised Skill Discovery for Agentic Data Analysiscited by 2
  10. 2025DSBC : Data Science task Benchmarking with Context engineeringcited by 1
  11. 2026Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguitycited by 1
  12. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  13. 2026VESTA: Visual Exploration with Statistical Tool Agentscited by 1
  14. 2026Before the Action: Benchmarking LLMs on Prospective Hypothesis Discoverycited by 1
  15. 2026DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?cited by 1
  16. 2025A Multi-Agent Approach to Neurological Clinical Reasoningcited by 0
  17. 2026COINBench: Moving Beyond Individual Perspectives to Collective Intent Understandingcited by 0
  18. 2026Structure-Grounded Knowledge Retrieval via Code Dependencies for Multi-Step Data Reasoningcited by 0
  19. 2026DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysiscited by 0
  20. 2026Text Analytics Evaluation Framework: A Case Study on LLMs and Social Mediacited by 0
  21. 2026Business Utility of Large Language Models as Exploratory Data Analysis Agentscited by 0
  22. 2026GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Sciencecited by 0
  23. 2026VeriGraph: Towards Verifiable Data-Analytic Agentscited by 0
  24. 2026Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluationcited by 0
  25. 2026UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Taskscited by 0
  26. 2026Fisher-R1: Training LLM Agents for Reliable Hypothesis Testingcited by 0
  27. 2026Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?cited by 0
  28. 2026Studying Without a Syllabus: Task-Agnostic Environment Preprocessingcited by 0
  29. 2026Skill-based Agentic Evaluation for Real-time Data Science Taskscited by 0
  30. 2026Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agentscited by 0
  31. 2026Fail Loudly: An Auditable Runtime for Agentic Data Analysiscited by 0
  32. 2026StateGuard: Analytical-State Management with Validity-Aware Intervention for Long-Horizon Data Agentscited by 0
  33. 2026Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflowscited by 0
  34. 2026Structured Composition of Verifiable Atomic Insights for Table-to-Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning
  2. 2025FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
  3. 2026FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents
  4. 2026DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
  5. 2025FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
  6. 2026Herculean: An Agentic Benchmark for Financial Intelligence
  7. 2026FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance
  8. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents