# Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab

arXiv: https://arxiv.org/abs/2507.02083 · HTML: https://mlsanity.com/paper/2507.02083/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2025.

## Built on by (most cited first, 12 of 12)

- [Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends](https://mlsanity.com/paper/2508.11548/) (2025)
- [Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models](https://mlsanity.com/paper/2608.09696/) (2026)
- [AInsteinBench: Benchmarking Coding Agents on Scientific Repositories](https://mlsanity.com/paper/2512.21373/) (2025)
- [ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments](https://mlsanity.com/paper/2609.19134/) (2026)
- [DiG-bench: Discovery in Games](https://mlsanity.com/paper/2608.12593/) (2026)
- [K-Bench: measuring model performance on real scientific agent requests](https://mlsanity.com/paper/2608.21601/) (2026)
- [Can LLMs Discover Scientific Laws in Real and Parallel Worlds?](https://mlsanity.com/paper/2609.01552/) (2026)
- [LLM Agents for Knowledge Discovery in Atomic Layer Processing](https://mlsanity.com/paper/2509.26201/) (2025)
- [On the post-hoc Evaluation of PDE Discovery: A Multifaceted Challenge of Scientific Advancement](https://mlsanity.com/paper/2607.23753/) (2026)
- [ChemWorld: Programmable Chemical Worlds for Controlled and Replayable Agent Experimentation](https://mlsanity.com/paper/2608.10792/) (2026)
- [WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents](https://mlsanity.com/paper/2609.27490/) (2026)
- [LLM-IDEA: Identifiability-Driven Experimental Agent for Autonomous Discovery of Mechanistic World Models](https://mlsanity.com/paper/2610.11253/) (2026)

## Abstract (arXiv)

Designing experiments and result interpretations are core scientific competencies, particularly in biology, where researchers perturb complex systems to uncover the underlying systems. Recent efforts to evaluate the scientific capabilities of large language models (LLMs) fail to test these competencies because wet-lab experimentation is prohibitively expensive: in expertise, time and equipment. We introduce SciGym, a first-in-class benchmark that assesses LLMs' iterative experiment design and analysis abilities in open-ended scientific discovery tasks. SciGym overcomes the challenge of wet-lab costs by running a dry lab of biological systems. These models, encoded in Systems Biology Markup Language, are efficient for generating simulated data, making them ideal testbeds for experimentation on realistically complex systems. We evaluated six frontier LLMs on 137 small systems, and released a total of 350 systems. Our evaluation shows that while more capable models demonstrated superior performance, all models' performance declined significantly as system complexity increased, suggesting substantial room for improvement in the scientific capabilities of LLM agents.