mlsanityData to 14 Oct 2026

BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science

arXiv 2505.07889 ↗First posted 11 May 2025cs.CLcited by 16

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols. To address this fundamental challenge, we present BioProBench, a comprehensive resource for procedural reasoning in biology. BioProBench is grounded in BioProCorpus, a foundational collection of 22,413 human-written protocols. From this corpus, we systematically constructed a dataset of 523,784 task instances, offering both a large-scale training resource and a rigorous benchmark with novel metrics. Evaluating 10 mainstream LLMs, we find that while general comprehension is high, performance drops significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness. To demonstrate the value of BioProCorpus in mitigating these issues, we developed ProAgent, grounded in our corpus, ProAgent substantially advances the state-of-the-art. https://github.com/YuyangSunshine/bioprobench and https://huggingface.co/BioProBench.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 26 of its 55 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

26 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Can large language models reason about medical questions?17 Jul

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2018Datasheets for Datasetscited by 1,386
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
13 more
  1. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  2. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  3. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
  4. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  5. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 100
  6. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  7. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  8. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 89
  9. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  10. 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 43
  11. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  12. 2024LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labscited by 25
  13. 2023ProBio: A Protocol-guided Multimodal Dataset for Molecular Biology Labcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  2. 2025CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoningcited by 5
  3. 2025Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberationcited by 4
  4. 2026SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?cited by 4
  5. 2025Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanismcited by 3
  6. 2026SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verificationcited by 2
  7. 2026K-Bench: measuring model performance on real scientific agent requestscited by 2
  8. 2026Memory Is Not Always Needed: Characterizing Conditional Memory in Scientific Reasoningcited by 1
  9. 2026BioPIE: A Biomedical Protocol Information Extraction Dataset for Experiment Understandingcited by 0
  10. 2026Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signalscited by 0
  11. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  12. 2026Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Modelscited by 0
4 more
  1. 2026BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modificationcited by 0
  2. 2026Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Modelscited by 0
  3. 2026Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Taskscited by 0
  4. 2026LabCompiler: A Framework for Verifiable Human-Machine Laboratory Execution from Imperfect Protocolscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Research
  2. 2026ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
  3. 2026BABE: Biology Arena BEnchmark
  4. 2025BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation
  5. 2025BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research
  6. 2026BioProAgent: Neuro-Symbolic Grounding for Constrained Scientific Planning
  7. 2026ProtoAct: Turning Wet-Lab Protocols into Embodied Robotic Actions
  8. 2026BioPhys-Bridge: A Benchmark for Interdisciplinary Scientific Reasoning in Physics-Grounded Biological Research