# BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science

arXiv: https://arxiv.org/abs/2505.07889 · HTML: https://mlsanity.com/paper/2505.07889/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 11 May 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine](https://mlsanity.com/paper/2308.09442/) (2023)
- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology](https://mlsanity.com/paper/2310.10632/) (2023)
- [A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks](https://mlsanity.com/paper/2310.04270/) (2023)
- [LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs](https://mlsanity.com/paper/2410.14182/) (2024)
- [ProBio: A Protocol-guided Multimodal Dataset for Molecular Biology Lab](https://mlsanity.com/paper/2311.00556/) (2023)

## Built on by (most cited first, 16 of 16)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning](https://mlsanity.com/paper/2510.13166/) (2025)
- [Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation](https://mlsanity.com/paper/2509.14760/) (2025)
- [SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?](https://mlsanity.com/paper/2605.21740/) (2026)
- [Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanism](https://mlsanity.com/paper/2510.15600/) (2025)
- [SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification](https://mlsanity.com/paper/2606.04579/) (2026)
- [K-Bench: measuring model performance on real scientific agent requests](https://mlsanity.com/paper/2608.21601/) (2026)
- [Memory Is Not Always Needed: Characterizing Conditional Memory in Scientific Reasoning](https://mlsanity.com/paper/2608.23982/) (2026)
- [BioPIE: A Biomedical Protocol Information Extraction Dataset for Experiment Understanding](https://mlsanity.com/paper/2601.04524/) (2026)
- [Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals](https://mlsanity.com/paper/2605.23384/) (2026)
- [Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches](https://mlsanity.com/paper/2606.01145/) (2026)
- [Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models](https://mlsanity.com/paper/2608.17202/) (2026)
- [BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification](https://mlsanity.com/paper/2608.23898/) (2026)
- [Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models](https://mlsanity.com/paper/2609.13005/) (2026)
- [Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks](https://mlsanity.com/paper/2610.00084/) (2026)
- [LabCompiler: A Framework for Verifiable Human-Machine Laboratory Execution from Imperfect Protocols](https://mlsanity.com/paper/2610.19550/) (2026)

## Abstract (arXiv)

The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols. To address this fundamental challenge, we present BioProBench, a comprehensive resource for procedural reasoning in biology. BioProBench is grounded in BioProCorpus, a foundational collection of 22,413 human-written protocols. From this corpus, we systematically constructed a dataset of 523,784 task instances, offering both a large-scale training resource and a rigorous benchmark with novel metrics. Evaluating 10 mainstream LLMs, we find that while general comprehension is high, performance drops significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness. To demonstrate the value of BioProCorpus in mitigating these issues, we developed ProAgent, grounded in our corpus, ProAgent substantially advances the state-of-the-art. https://github.com/YuyangSunshine/bioprobench and https://huggingface.co/BioProBench.