# Science sandboxes measure the scientific capability of AI agents

arXiv: https://arxiv.org/abs/2608.30165 · HTML: https://mlsanity.com/paper/2608.30165/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Empowering Biomedical Discovery with AI Agents](https://mlsanity.com/paper/2404.02831/) (2024)
- [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Design](https://mlsanity.com/paper/2502.14944/) (2025)
- [Life After Benchmark Saturation: A Case Study of CORE-Bench](https://mlsanity.com/paper/2606.26158/) (2026)
- [Iterative Foundation Model Fine-Tuning on Multiple Rewards](https://mlsanity.com/paper/2511.00220/) (2025)

## Built on by (most cited first, 1 of 1)

- [WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents](https://mlsanity.com/paper/2609.27490/) (2026)

## Abstract (arXiv)

Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.