# Olympus: a benchmarking framework for noisy optimization and experiment planning

arXiv: https://arxiv.org/abs/2010.04153 · HTML: https://mlsanity.com/paper/2010.04153/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 29 of 29)

- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Benchmarking the Performance of Bayesian Optimization across Multiple Experimental Materials Science Domains](https://mlsanity.com/paper/2106.01309/) (2021)
- [GAUCHE: A Library for Gaussian Processes in Chemistry](https://mlsanity.com/paper/2212.04450/) (2022)
- [MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learning](https://mlsanity.com/paper/2310.08252/) (2023)
- [Posterior Sampling-Based Bayesian Optimization with Tighter Bayesian Regret Bounds](https://mlsanity.com/paper/2311.03760/) (2023)
- [Large language models as uncertainty-calibrated optimizers for experimental discovery](https://mlsanity.com/paper/2504.06265/) (2025)
- [LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs](https://mlsanity.com/paper/2605.24043/) (2026)
- [Bayesian optimization with known experimental and design constraints for chemistry applications](https://mlsanity.com/paper/2203.17241/) (2022)
- [Machine Learning with Knowledge Constraints for Process Optimization of Open-Air Perovskite Solar Cell Manufacturing](https://mlsanity.com/paper/2110.01387/) (2021)
- [Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling](https://mlsanity.com/paper/2505.23913/) (2025)
- [Unleashing LLMs in Bayesian Optimization: Preference-Guided Framework for Scientific Discovery](https://mlsanity.com/paper/2605.17976/) (2026)
- [Golem: An algorithm for robust experiment and process optimization](https://mlsanity.com/paper/2103.03716/) (2021)
- [Graph Machine Learning for Design of High-Octane Fuels](https://mlsanity.com/paper/2206.00619/) (2022)
- [Gemini: Dynamic Bias Correction for Autonomous Experimentation and Molecular Simulation](https://mlsanity.com/paper/2103.03391/) (2021)
- [Randomized Kriging Believer for Parallel Bayesian Optimization with Regret Bounds](https://mlsanity.com/paper/2603.01470/) (2026)
- [BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks](https://mlsanity.com/paper/2605.17000/) (2026)
- [Evolutionary Intelligence for Scientific Discovery: From Evolutionary Computation to Cumulative Discovery Systems](https://mlsanity.com/paper/2607.09025/) (2026)
- [Opportunities for Machine Learning to Accelerate Halide Perovskite Commercialization and Scale-Up](https://mlsanity.com/paper/2110.03923/) (2021)
- [Bayesian Optimization for General Reaction Conditions](https://mlsanity.com/paper/2502.18966/) (2025)
- [CatBOX: A Categorical-Continuous Bayesian Optimization with Spectral Mixture Kernels for Accelerated Catalysis Experiments](https://mlsanity.com/paper/2505.17393/) (2025)
- [A Critical Examination of Active Learning Workflows in Materials Science](https://mlsanity.com/paper/2601.05946/) (2026)
- [Information Theoretic Bayesian Optimization over the Probability Simplex](https://mlsanity.com/paper/2603.09793/) (2026)
- [Towards Regret Guarantees for One-Step Lookahead Bayesian Optimization](https://mlsanity.com/paper/2606.00956/) (2026)
- [BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems](https://mlsanity.com/paper/2608.02612/) (2026)
- [ChemWorld: Programmable Chemical Worlds for Controlled and Replayable Agent Experimentation](https://mlsanity.com/paper/2608.10792/) (2026)
- [Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation](https://mlsanity.com/paper/2609.36788/) (2026)
- [IQS-BO: In-Context Query Selection for Bayesian Optimisation](https://mlsanity.com/paper/2610.01269/) (2026)
- [Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings](https://mlsanity.com/paper/2610.01827/) (2026)
- [Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery](https://mlsanity.com/paper/2610.18449/) (2026)

## Abstract (arXiv)

Research challenges encountered across science, engineering, and economics can frequently be formulated as optimization tasks. In chemistry and materials science, recent growth in laboratory digitization and automation has sparked interest in optimization-guided autonomous discovery and closed-loop experimentation. Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms to achieve desired experimentation goals with the minimum number of trials. However, the experiment planning strategy that is most suitable to a scientific discovery task is a priori unknown while rigorous comparisons of different strategies are highly time and resource demanding. As optimization algorithms are typically benchmarked on low-dimensional synthetic functions, it is unclear how their performance would translate to noisy, higher-dimensional experimental tasks encountered in chemistry and materials science. We introduce Olympus, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms against realistic experiments emulated via probabilistic deep-learning models. Olympus includes a collection of experimentally derived benchmark sets from chemistry and materials science and a suite of experiment planning strategies that can be easily accessed via a user-friendly python interface. Furthermore, Olympus facilitates the integration, testing, and sharing of custom algorithms and user-defined datasets. In brief, Olympus mitigates the barriers associated with benchmarking optimization algorithms on realistic experimental scenarios, promoting data sharing and the creation of a standard framework for evaluating the performance of experiment planning strategies