# LLMs Outperform Experts on Challenging Biology Benchmarks

arXiv: https://arxiv.org/abs/2505.06108 · HTML: https://mlsanity.com/paper/2505.06108/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [CRISPR-GPT for Agentic Automation of Gene-editing Experiments](https://mlsanity.com/paper/2404.18021/) (2024)
- [BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology](https://mlsanity.com/paper/2310.10632/) (2023)
- [ChatBench: From Static Benchmarks to Human-AI Evaluation](https://mlsanity.com/paper/2504.07114/) (2025)
- [Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark](https://mlsanity.com/paper/2504.16137/) (2025)
- [MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants](https://mlsanity.com/paper/2412.12661/) (2024)
- [Enterprise Benchmarks for Large Language Model Evaluation](https://mlsanity.com/paper/2410.12857/) (2024)

## Built on by (most cited first, 9 of 9)

- [When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation](https://mlsanity.com/paper/2602.16763/) (2026)
- [STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports](https://mlsanity.com/paper/2508.09853/) (2025)
- [Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology](https://mlsanity.com/paper/2602.16703/) (2026)
- [SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?](https://mlsanity.com/paper/2604.10718/) (2026)
- [Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution](https://mlsanity.com/paper/2603.05308/) (2026)
- [MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering](https://mlsanity.com/paper/2605.12361/) (2026)
- [Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models](https://mlsanity.com/paper/2606.05531/) (2026)
- [CLINB: A Climate Intelligence Benchmark for Foundational Models](https://mlsanity.com/paper/2511.11597/) (2025)
- [Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches](https://mlsanity.com/paper/2606.01145/) (2026)

## Abstract (arXiv)

This study systematically evaluates 27 frontier Large Language Models on eight biology benchmarks spanning molecular biology, genetics, cloning, virology, and biosecurity. Models from major AI developers released between November 2022 and April 2025 were assessed through ten independent runs per benchmark. The findings reveal dramatic improvements in biological capabilities. Top model performance increased more than 4-fold on the challenging text-only subset of the Virology Capabilities Test over the study period, with OpenAI's o3 now performing twice as well as expert virologists. Several models now match or exceed expert-level performance on other challenging benchmarks, including the biology subsets of GPQA and WMDP and LAB-Bench CloningScenarios. Contrary to expectations, chain-of-thought did not substantially improve performance over zero-shot evaluation, while extended reasoning features in o3-mini and Claude 3.7 Sonnet typically improved performance as predicted by inference scaling. Benchmarks such as PubMedQA and the MMLU and WMDP biology subsets exhibited performance plateaus well below 100%, suggesting benchmark saturation and errors in the underlying benchmark data. The analysis highlights the need for more sophisticated evaluation methodologies as AI systems continue to advance.