# Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2207.00479 · HTML: https://mlsanity.com/paper/2207.00479/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Builds on (most cited first, 16 of 16)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Batch Bayesian Optimization via Local Penalization](https://mlsanity.com/paper/1505.08052/) (2015)
- [Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space](https://mlsanity.com/paper/1706.01825/) (2017)
- [AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification](https://mlsanity.com/paper/2110.13511/) (2021)
- [Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation](https://mlsanity.com/paper/1901.10452/) (2019)
- [mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions](https://mlsanity.com/paper/1703.03373/) (2017)
- [Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019](https://mlsanity.com/paper/2201.03801/) (2022)
- [Fully Distributed Bayesian Optimization with Stochastic Policies](https://mlsanity.com/paper/1902.09992/) (2019)

## Built on by (most cited first, 6 of 6)

- [Optimizing Distributed Training on Frontier for Large Language Models](https://mlsanity.com/paper/2312.12705/) (2023)
- [How does ion temperature gradient turbulence depend on magnetic geometry? Insights from data and machine learning](https://mlsanity.com/paper/2502.11657/) (2025)
- [Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach](https://mlsanity.com/paper/2404.05768/) (2024)
- [When Does Global Attention Help? A Unified Empirical Study on Atomistic Graph Learning](https://mlsanity.com/paper/2510.05583/) (2025)
- [Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimization](https://mlsanity.com/paper/2603.13501/) (2026)
- [mlr3mbo: Bayesian Optimization in R](https://mlsanity.com/paper/2603.29730/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) is a promising approach for hyperparameter optimization of deep neural networks (DNNs), where each model training can take minutes to hours. In BO, a computationally cheap surrogate model is employed to learn the relationship between parameter configurations and their performance such as accuracy. Parallel BO methods often adopt single manager/multiple workers strategies to evaluate multiple hyperparameter configurations simultaneously. Despite significant hyperparameter evaluation time, the overhead in such centralized schemes prevents these methods to scale on a large number of workers. We present an asynchronous-decentralized BO, wherein each worker runs a sequential BO and asynchronously communicates its results through shared storage. We scale our method without loss of computational efficiency with above 95% of worker's utilization to 1,920 parallel workers (full production queue of the Polaris supercomputer) and demonstrate improvement in model accuracy as well as faster convergence on the CANDLE benchmark from the Exascale computing project.