mlsanityData to 14 Oct 2026

BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology

arXiv 2503.00096 ↗First posted 28 Feb 2025q-bio.QM cs.AIcited by 89

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and rote knowledge tasks, towards more practical work such as literature review and experimental planning. Bioinformatics is a domain where fully autonomous AI-driven discovery may be near, but no extensive benchmarks for measuring progress have been introduced to date. We therefore present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis with nearly 300 associated open-answer questions designed to measure the ability of LLM-based agents to explore biological datasets, perform long, multi-step analytical trajectories, and interpret the nuanced results of those analyses. We evaluate the performance of two frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent framework we open source. We find that even the latest frontier models only achieve 17% accuracy in the open-answer regime, and no better than random in a multiple-choice setting. By exposing the current limitations of frontier models, we hope BixBench can spur the development of agents capable of conducting rigorous bioinformatic analysis and accelerate scientific discovery.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 34 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

24 of 34 references.

References · 24

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  6. 2023Large Language Modelscited by 780
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  8. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  9. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  10. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  11. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  12. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
12 more
  1. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  2. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  3. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  4. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  5. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 109
  6. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 77
  7. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 65
  8. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 58
  9. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  10. 2024Aviary: training language agents on challenging scientific taskscited by 26
  11. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 25
  12. 2024An Evaluation of Large Language Models in Bioinformatics Researchcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 28; 2026: 61.

89 papers.

Sort:

By area:

  1. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
  2. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  3. 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 92
  4. 2025Robin: A multi-agent system for automating scientific discoverycited by 62
  5. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  6. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  7. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 33
  8. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 24
  9. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 23
  10. 2025BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Sciencecited by 16
  11. 2025PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priorscited by 15
  12. 2025LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Sciencecited by 15
38 more
  1. 2026BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarkscited by 15
  2. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 13
  3. 2026HeurekaBench: A Benchmarking Framework for AI Co-scientistcited by 12
  4. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 12
  5. 2026Benchmarking AI Agents for Addressing Scientific Challenges Across Scalescited by 12
  6. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11
  7. 2025ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?cited by 11
  8. 2026BioAgent Bench: An AI Agent Evaluation Suite for Bioinformaticscited by 10
  9. 2026DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinkingcited by 10
  10. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  11. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Researchcited by 8
  12. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  13. 2025Benchmarking AI scientists for omics data driven biological discoverycited by 7
  14. 2025K-Dense Analyst: Towards Fully Automated Scientific Analysiscited by 6
  15. 2025MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncologycited by 6
  16. 2026Harnessing AtomisticSkills for Agentic Atomistic Researchcited by 6
  17. 2026ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecuritycited by 6
  18. 2025CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamicscited by 5
  19. 2026BixBench3: Benchmarking AI agents on research-study-scale computational biology taskscited by 5
  20. 2025Offline Learning and Forgetting for Reasoning with Large Language Modelscited by 4
  21. 2025CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researcherscited by 4
  22. 2026scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discoverycited by 4
  23. 2026MDGYM: Benchmarking AI Agents on Molecular Simulationscited by 4
  24. 2026SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?cited by 4
  25. 2026SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysiscited by 4
  26. 2025CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Researchcited by 3
  27. 2025ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challengescited by 3
  28. 2026CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Computecited by 3
  29. 2026SciNav: A General Agent Framework for Scientific Coding Taskscited by 3
  30. 2026Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agentscited by 3
  31. 2026AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agentscited by 3
  32. 2026Verifiable Benchmarking of Long-Horizon Spatial Biologycited by 3
  33. 2026Locating Hidden Failures Makes Long-Horizon Agents More Reliablecited by 3
  34. 2025Benchmarking LLM-based agents for single-cell omics analysiscited by 2
  35. 2026Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplerscited by 2
  36. 2026K-Bench: measuring model performance on real scientific agent requestscited by 2
  37. 2025Speak to a Protein: An Interactive Multimodal Co-Scientistcited by 1
  38. 2026SciDesignBench: Benchmarking and Improving Language Models for Scientific Inverse Designcited by 1

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systems
  2. 2026The limits of bio-molecular modeling with large language models : a cross-scale evaluation
  3. 2026Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
  4. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  5. 2025BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation
  6. 2024GenoTEX: An LLM Agent Benchmark for Automated Gene Expression Data Analysis
  7. 2026BABE: Biology Arena BEnchmark
  8. 2025LLM4Cell: A Survey of Large Language and Agentic Models for Single-Cell Biology