mlsanityData to 7 Oct 2026

LAB-Bench: Measuring Capabilities of Language Models for Biology Research

arXiv 2407.10362 ↗First posted 14 Jul 2024cs.AIcited by 138

Not on the map. Why?

Cited by 48 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is widespread optimism that frontier Large Language Models (LLMs) and LLM-augmented systems have the potential to rapidly accelerate scientific discovery across disciplines. Today, many benchmarks exist to measure LLM knowledge and reasoning on textbook-style science questions, but few if any benchmarks are designed to evaluate language model performance on practical tasks required for scientific research, such as literature search, protocol planning, and data analysis. As a step toward building such benchmarks, we introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities, including recall and reasoning over literature, interpretation of figures, access and navigation of databases, and comprehension and manipulation of DNA and protein sequences. Importantly, in contrast to previous scientific benchmarks, we expect that an AI system that can achieve consistently high scores on the more difficult LAB-Bench tasks would serve as a useful assistant for researchers in areas such as literature search and molecular cloning. As an initial assessment of the emergent scientific task capabilities of frontier language models, we measure performance of several against our benchmark and report results compared to human expert biology researchers. We will continue to update and expand LAB-Bench over time, and expect it to serve as a useful tool in the development of automated research systems going forward. A public subset of LAB-Bench is available for use at the following URL: https://huggingface.co/datasets/futurehouse/lab-bench

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 12 of its 48 references that are in the map. First posted on arXiv 14 Jul 2024.

Builds on

12 of 48 references.

References · 12

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  5. 2023Large Language Modelscited by 774
  6. 2022Galactica: A Large Language Model for Sciencecited by 736
  7. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  8. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  9. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  11. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  12. 2021Machine Learning with a Reject Option: A surveycited by 107

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 57; 2026: 73.

138 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2024OpenAI o1 System Cardcited by 1,220
  4. 2025Humanity's Last Examcited by 467
  5. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  6. 2025Survey on Evaluation of LLM-based Agentscited by 190
  7. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  8. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  9. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  10. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 74
  11. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 58
  12. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
38 more
  1. 2025Visual Agentic AI for Spatial Reasoning with a Dynamic APIcited by 42
  2. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  3. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  5. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  6. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 30
  7. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  8. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  9. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
  10. 2024Aviary: training language agents on challenging scientific taskscited by 23
  11. 2025MDCrow: Automating Molecular Dynamics Workflows with Large Language Modelscited by 22
  12. 2025ToolUniverse: An open platform for democratizing AI scientistscited by 22
  13. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  14. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 17
  15. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
  16. 2025Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanismcited by 16
  17. 2025BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Sciencecited by 14
  18. 2025Towards LLM Agents for Earth Observationcited by 13
  19. 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
  20. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  21. 2025Contextualizing biological perturbation experiments through languagecited by 10
  22. 2025STELLA: Self-Evolving LLM Agent for Biomedical Researchcited by 10
  23. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  24. 2025ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?cited by 10
  25. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  26. 2024HELM: Hierarchical Encoding for mRNA Language Modelingcited by 9
  27. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  28. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  29. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 9
  30. 2026HeurekaBench: A Benchmarking Framework for AI Co-scientistcited by 9
  31. 2026Prescriptive Scaling Laws for Data Constrained Trainingcited by 9
  32. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  33. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  34. 2026BioAgent Bench: An AI Agent Evaluation Suite for Bioinformaticscited by 8
  35. 2025Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learningcited by 6
  36. 2025A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyondcited by 6
  37. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Researchcited by 6
  38. 2026Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproductioncited by 6

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab
  2. 2026BABE: Biology Arena BEnchmark
  3. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  4. 2025Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting
  5. 2024SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
  6. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents
  7. 2025RoBiologyDataChoiceQA: A Romanian Dataset for improving Biology understanding of Large Language Models
  8. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science