# BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research arXiv: https://arxiv.org/abs/2505.16100 ยท HTML: https://mlsanity.com/paper/2505.16100/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. - Builds on 26 of its 36 references that are in the map. - First posted on arXiv 22 May 2025. ## Placed in - LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/) ## Builds on (most cited first, 26 of 26) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024) - [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022) - [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025) - [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023) - [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023) - [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024) - [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024) - [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024) - [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024) - [Empowering Biomedical Discovery with AI Agents](https://mlsanity.com/paper/2404.02831/) (2024) - [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024) - [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024) - [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024) - [AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions](https://mlsanity.com/paper/2410.20424/) (2024) - [Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?](https://mlsanity.com/paper/2407.10956/) (2024) - [BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments](https://mlsanity.com/paper/2405.17631/) (2024) - [Automated Hypothesis Validation with Agentic Sequential Falsifications](https://mlsanity.com/paper/2502.09858/) (2025) - [Data-driven Discovery with Large Generative Models](https://mlsanity.com/paper/2402.13610/) (2024) - [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023) - [Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performance](https://mlsanity.com/paper/2411.03562/) (2024) - [Large Language Models Streamline Automated Machine Learning for Clinical Studies](https://mlsanity.com/paper/2308.14120/) (2023) - [A foundation model for human-AI collaboration in medical literature mining](https://mlsanity.com/paper/2501.16255/) (2025) - [BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systems](https://mlsanity.com/paper/2501.06314/) (2025) ## Built on by (most cited first, 8 of 8) - [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025) - [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025) - [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025) - [AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery](https://mlsanity.com/paper/2605.23204/) (2026) - [CausalGame: Benchmarking Causal Thinking of LLM Agents in Games](https://mlsanity.com/paper/2607.04293/) (2026) - [Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging](https://mlsanity.com/paper/2607.10789/) (2026) - [Plato-Bio: verification-first biological novelty screening with temporal rediscovery and structural benchmarks](https://mlsanity.com/paper/2607.23975/) (2026) - [BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies](https://mlsanity.com/paper/2610.07614/) (2026) ## Abstract (arXiv) Validating scientific hypotheses is a central challenge in biomedical research, and remains difficult for artificial intelligence (AI) agents due to the complexity of real-world data analysis and evidence interpretation. In this work, we present BioDSA-1K, a benchmark designed to evaluate AI agents on realistic, data-driven biomedical hypothesis validation tasks. BioDSA-1K consists of 1,029 hypothesis-centric tasks paired with 1,177 analysis plans, curated from over 300 published biomedical studies to reflect the structure and reasoning found in authentic research workflows. Each task includes a structured hypothesis derived from the original study's conclusions, expressed in the affirmative to reflect the language of scientific reporting, and one or more pieces of supporting evidence grounded in empirical data tables. While these hypotheses mirror published claims, they remain testable using standard statistical or machine learning methods. The benchmark enables evaluation along four axes: (1) hypothesis decision accuracy, (2) alignment between evidence and conclusion, (3) correctness of the reasoning process, and (4) executability of the AI-generated analysis code. Importantly, BioDSA-1K includes non-verifiable hypotheses: cases where the available data are insufficient to support or refute a claim, reflecting a common yet underexplored scenario in real-world science. We propose BioDSA-1K as a foundation for building and evaluating generalizable, trustworthy AI agents for biomedical discovery.