# Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery

arXiv: https://arxiv.org/abs/2610.18449 · HTML: https://mlsanity.com/paper/2610.18449/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 30 of 31)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [MLGym: A New Framework and Benchmark for Advancing AI Research Agents](https://mlsanity.com/paper/2502.14499/) (2025)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents](https://mlsanity.com/paper/2406.06769/) (2024)
- [Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning](https://mlsanity.com/paper/1903.04703/) (2019)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments](https://mlsanity.com/paper/2405.17631/) (2024)
- [Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading](https://mlsanity.com/paper/2607.08964/) (2026)
- [Olympus: a benchmarking framework for noisy optimization and experiment planning](https://mlsanity.com/paper/2010.04153/) (2020)
- [BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery](https://mlsanity.com/paper/2501.01540/) (2025)
- [A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimization](https://mlsanity.com/paper/2505.16094/) (2025)
- [LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs](https://mlsanity.com/paper/2605.24043/) (2026)
- [Benchmarking Active Learning Strategies for Materials Optimization and Discovery](https://mlsanity.com/paper/2204.05838/) (2022)
- [POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimization](https://mlsanity.com/paper/2509.21737/) (2025)
- [MADE: Benchmark Environments for Closed-Loop Materials Discovery](https://mlsanity.com/paper/2601.20996/) (2026)
- [MolMem: Memory-Augmented Agentic Reinforcement Learning for Sample-Efficient Molecular Optimization](https://mlsanity.com/paper/2604.12237/) (2026)
- [Purely Agent-Driven Black-Box Optimization for Biological Design](https://mlsanity.com/paper/2601.22382/) (2026)
- [La Agente 'Optima: Towards Agentic Self-Driving Laboratories](https://mlsanity.com/paper/2609.04564/) (2026)

## Abstract (arXiv)

AI agents for autonomous scientific discovery must decide which experiments to run under a limited budget. Most benchmarks for science agents provide fixed datasets, while others that support active experimentation largely rely on simulation. We introduce Autonomous Science Bench, an initial release comprising six budgeted design-build-test-learn campaigns grounded in published experiments across biology, chemistry, and materials science. Agents select batches of designs, receive measurements from a lab service, and use the results to guide subsequent experiments before delivering a predictive model. Success requires passing two calibrated gates: a discovery gate, which evaluates the designs the agent measured, and a learning gate, which evaluates its model's ranking of held-out designs. In a pilot evaluation of eight agent-model configurations and 144 trials, the best configuration succeeds in 67% of trials, while the hardest campaign is solved in only three of 24 attempts. The two gates reveal different weaknesses: some agents build accurate rankings but find too few promising designs, while others find promising designs but fail to predict which unseen designs will perform well. These differences are obscured by overall success rates alone. We release version 0.1 of the campaigns, lab runtime, and public leaderboard at https://yibow.me/autonomous-science-bench.