# BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies

arXiv: https://arxiv.org/abs/2610.07614 · HTML: https://mlsanity.com/paper/2610.07614/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 17 of 17)

- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024)
- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024)
- [AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge](https://mlsanity.com/paper/2412.13670/) (2024)
- [FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights](https://mlsanity.com/paper/2602.02905/) (2026)
- [ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?](https://mlsanity.com/paper/2510.24591/) (2025)
- [Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs](https://mlsanity.com/paper/2510.02340/) (2025)
- [BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research](https://mlsanity.com/paper/2505.16100/) (2025)
- [Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments](https://mlsanity.com/paper/2601.07606/) (2026)
- [BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks](https://mlsanity.com/paper/2608.25286/) (2026)
- [Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases](https://mlsanity.com/paper/2603.22767/) (2026)
- [Can AI Agents Synthesize Scientific Conclusions?](https://mlsanity.com/paper/2606.11337/) (2026)

## Abstract (arXiv)

We evaluate whether AI agents can match the reported findings of published biomedical studies using public data. Existing evaluations do not consistently separate analysis from prior knowledge or retrieval of the published answer. We introduce BioStudyBench, a benchmark of 25 long-horizon analysis tasks drawn from studies first published between July and September 2026, after the developer-reported knowledge cutoffs of the models we evaluate, semi-automatically filtered down from 404,019 PubMed records. In each task, the agent receives a neutral research question but no data files, so it must find and download the relevant public data, search the literature through tools that return only records dated before its cutoff, and report findings through data analysis. To measure gains over prior knowledge, we run every task both with and without access to data and tools. Across eight models, access to data and tools raises the pass rate by 47 percentage points on average over the no-data baseline. Open-weight models across sizes trail closed-weight models, with the best open-weight model passing 81.3% of tasks against 94.7% for the best closed-weight model.