# BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology

arXiv: https://arxiv.org/abs/2503.00096 · HTML: https://mlsanity.com/paper/2503.00096/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 24 of 24)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Empowering Biomedical Discovery with AI Agents](https://mlsanity.com/paper/2404.02831/) (2024)
- [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024)
- [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024)
- [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://mlsanity.com/paper/2311.09835/) (2023)
- [DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models](https://mlsanity.com/paper/2410.07331/) (2024)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023)
- [An Evaluation of Large Language Models in Bioinformatics Research](https://mlsanity.com/paper/2402.13714/) (2024)

## Built on by (most cited first, 30 of 89)

- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Kosmos: An AI Scientist for Autonomous Discovery](https://mlsanity.com/paper/2511.02824/) (2025)
- [Robin: A multi-agent system for automating scientific discovery](https://mlsanity.com/paper/2505.13400/) (2025)
- [EXP-Bench: Can AI Conduct AI Research Experiments?](https://mlsanity.com/paper/2505.24785/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [Paper2Video: Automatic Video Generation from Scientific Papers](https://mlsanity.com/paper/2510.05096/) (2025)
- [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025)
- [BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science](https://mlsanity.com/paper/2505.07889/) (2025)
- [PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors](https://mlsanity.com/paper/2507.15550/) (2025)
- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks](https://mlsanity.com/paper/2604.24955/) (2026)
- [DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use](https://mlsanity.com/paper/2603.11076/) (2026)
- [HeurekaBench: A Benchmarking Framework for AI Co-scientist](https://mlsanity.com/paper/2601.01678/) (2026)
- [Automated Benchmark Auditing for AI Agents and Large Language Models](https://mlsanity.com/paper/2605.26079/) (2026)
- [Benchmarking AI Agents for Addressing Scientific Challenges Across Scales](https://mlsanity.com/paper/2606.12736/) (2026)
- [AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists](https://mlsanity.com/paper/2506.08140/) (2025)
- [ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?](https://mlsanity.com/paper/2510.24591/) (2025)
- [BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics](https://mlsanity.com/paper/2601.21800/) (2026)
- [DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking](https://mlsanity.com/paper/2605.26087/) (2026)
- [LLMs Outperform Experts on Challenging Biology Benchmarks](https://mlsanity.com/paper/2505.06108/) (2025)
- [LABBench2: An Improved Benchmark for AI Systems Performing Biology Research](https://mlsanity.com/paper/2604.09554/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Benchmarking AI scientists for omics data driven biological discovery](https://mlsanity.com/paper/2505.08341/) (2025)
- [K-Dense Analyst: Towards Fully Automated Scientific Analysis](https://mlsanity.com/paper/2508.07043/) (2025)
- [MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology](https://mlsanity.com/paper/2511.20490/) (2025)
- [Harnessing AtomisticSkills for Agentic Atomistic Research](https://mlsanity.com/paper/2605.24002/) (2026)
- [ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity](https://mlsanity.com/paper/2606.11150/) (2026)
- [CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics](https://mlsanity.com/paper/2509.20374/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and rote knowledge tasks, towards more practical work such as literature review and experimental planning. Bioinformatics is a domain where fully autonomous AI-driven discovery may be near, but no extensive benchmarks for measuring progress have been introduced to date. We therefore present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis with nearly 300 associated open-answer questions designed to measure the ability of LLM-based agents to explore biological datasets, perform long, multi-step analytical trajectories, and interpret the nuanced results of those analyses. We evaluate the performance of two frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent framework we open source. We find that even the latest frontier models only achieve 17% accuracy in the open-answer regime, and no better than random in a multiple-choice setting. By exposing the current limitations of frontier models, we hope BixBench can spur the development of agents capable of conducting rigorous bioinformatic analysis and accelerate scientific discovery.