# BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks

arXiv: https://arxiv.org/abs/2608.25286 · HTML: https://mlsanity.com/paper/2608.25286/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 83 references that are in the map.
- First posted on arXiv 26 Aug 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 16 of 16)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Empowering Biomedical Discovery with AI Agents](https://mlsanity.com/paper/2404.02831/) (2024)
- [DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents](https://mlsanity.com/paper/2406.06769/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?](https://mlsanity.com/paper/2606.07682/) (2026)
- [Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models](https://mlsanity.com/paper/2606.07157/) (2026)
- [LABBench2: An Improved Benchmark for AI Systems Performing Biology Research](https://mlsanity.com/paper/2604.09554/) (2026)
- [GenoTEX: An LLM Agent Benchmark for Automated Gene Expression Data Analysis](https://mlsanity.com/paper/2406.15341/) (2024)
- [Verifiable Benchmarking of Long-Horizon Spatial Biology](https://mlsanity.com/paper/2605.28065/) (2026)

## Built on by (most cited first, 5 of 5)

- [DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?](https://mlsanity.com/paper/2609.33357/) (2026)
- [BIABench: Evaluating AI agents on real-world bioimage analysis tasks](https://mlsanity.com/paper/2609.34274/) (2026)
- [Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists](https://mlsanity.com/paper/2610.04915/) (2026)
- [BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies](https://mlsanity.com/paper/2610.07614/) (2026)
- [ReproBAIT: Can coding agents build their own biological AI tools?](https://mlsanity.com/paper/2610.19810/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) promises to accelerate biological research by automating computational analyses. Yet the ability of AI agents to execute on computational biology at the scale of complete research studies has not been systematically evaluated. Here we introduce BixBench3, a benchmark that measures the capacity of AI agents to process raw biological data through to scientific results. We designed BixBench3 tasks to mirror the delegation of work from a scientist to an agent: the scientist chooses the research question and high-level methods, then delegates implementation of all analyses to the agent. In each task, an agent receives a research objective, methodological guidance, and raw data derived from a published scientific study, and must execute a sequence of analyses to achieve the research objective. The data artifacts resulting from these analyses, such as peak call matrices or differential expression tables, are programmatically graded against the corresponding artifacts generated and reported in the original study. Across 20 BixBench3 tasks encompassing the generation of 138 unique artifacts, we find that 13 frontier models achieve scores ranging from 0.00 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol. Agents perform worse on tasks with larger raw datasets (0.36 on tasks with <100 GB versus 0.10 on tasks with >100 GB) and on analyses requiring more sequential steps (0.36 at 1-2 steps vs 0.24 at 3+). On average, agents use 6.8 hours, 102 million tokens, and 43 to complete each task, with the longest attempts consuming 24 hours, 1.07 billion tokens, and 525. Notably, the highest-scoring agents used fewer tokens and were cheaper than less performant options. These results reveal that LLMs vary substantially in their ability to (1) execute multiple sequential analysis steps coherently, (2) manage large quantities of raw data, and (3) work across scientific domains.