# DSGym: A Holistic Framework for Evaluating and Training Data Science Agents arXiv: https://arxiv.org/abs/2601.16344 ยท HTML: https://mlsanity.com/paper/2601.16344/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. - Builds on 23 of its 42 references that are in the map. - First posted on arXiv 22 Jan 2026. ## Placed in - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 23 of 23) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024) - [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024) - [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024) - [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022) - [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025) - [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024) - [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023) - [MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization](https://mlsanity.com/paper/2402.11453/) (2024) - [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022) - [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024) - [AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions](https://mlsanity.com/paper/2410.20424/) (2024) - [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024) - [DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models](https://mlsanity.com/paper/2410.07331/) (2024) - [DataSciBench: An LLM Agent Benchmark for Data Science](https://mlsanity.com/paper/2502.13897/) (2025) - [DABstep: Data Agent Benchmark for Multi-step Reasoning](https://mlsanity.com/paper/2506.23719/) (2025) - [A Survey on Large Language Model-based Agents for Statistics and Data Science](https://mlsanity.com/paper/2412.14222/) (2024) - [Scaling Generalist Data-Analytic Agents](https://mlsanity.com/paper/2509.25084/) (2025) - [DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automation](https://mlsanity.com/paper/2503.07044/) (2025) - [LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions](https://mlsanity.com/paper/2510.04023/) (2025) - [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025) ## Built on by (most cited first, 15 of 15) - [Automated Benchmark Auditing for AI Agents and Large Language Models](https://mlsanity.com/paper/2605.26079/) (2026) - [InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem](https://mlsanity.com/paper/2602.14367/) (2026) - [Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis](https://mlsanity.com/paper/2604.24198/) (2026) - [Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search](https://mlsanity.com/paper/2603.01692/) (2026) - [Sanity Checks for Agentic Data Science](https://mlsanity.com/paper/2604.11003/) (2026) - [LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis](https://mlsanity.com/paper/2605.30434/) (2026) - [Unsupervised Skill Discovery for Agentic Data Analysis](https://mlsanity.com/paper/2606.06416/) (2026) - [OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents](https://mlsanity.com/paper/2606.26350/) (2026) - [An AI Co-Data-Scientist for Prioritizing Candidate Biomarkers from Wearable Sensor Data](https://mlsanity.com/paper/2604.14615/) (2026) - [D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery](https://mlsanity.com/paper/2604.27977/) (2026) - [Agentic-imodels: Evolving agentic interpretability tools via autoresearch](https://mlsanity.com/paper/2605.03808/) (2026) - [Data Journalist Agent: Transforming Data into Verifiable Multimodal Stories](https://mlsanity.com/paper/2606.11176/) (2026) - [GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science](https://mlsanity.com/paper/2606.16000/) (2026) - [Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System](https://mlsanity.com/paper/2606.24839/) (2026) - [Skill-based Agentic Evaluation for Real-time Data Science Tasks](https://mlsanity.com/paper/2609.16487/) (2026) ## Abstract (arXiv) Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.