# Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

arXiv: https://arxiv.org/abs/1912.05686 · HTML: https://mlsanity.com/paper/1912.05686/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2019.

## Built on by (most cited first, 6 of 6)

- [BoFire: Bayesian Optimization Framework Intended for Real Experiments](https://mlsanity.com/paper/2408.05040/) (2024)
- [Bayesian optimized deep ensemble for uncertainty quantification of deep neural networks: a system safety case study on sodium fast reactor thermal stratification modeling](https://mlsanity.com/paper/2412.08776/) (2024)
- [Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Models](https://mlsanity.com/paper/2106.00120/) (2021)
- [Evaluating Privacy Leakage in Split Learning](https://mlsanity.com/paper/2305.12997/) (2023)
- [Hypergraph: A Unified and Uniform Definition with Application to Chemical Hypergraph and More](https://mlsanity.com/paper/2405.12235/) (2024)
- [Automating Data-Driven Modeling and Analysis for Engineering Applications using Large Language Model Agents](https://mlsanity.com/paper/2510.01398/) (2025)

## Abstract (arXiv)

Deep learning models are full of hyperparameters, which are set manually before the learning process can start. To find the best configuration for these hyperparameters in such a high dimensional space, with time-consuming and expensive model training / validation, is not a trivial challenge. Bayesian optimization is a powerful tool for the joint optimization of hyperparameters, efficiently trading off exploration and exploitation of the hyperparameter space. In this paper, we discuss Bayesian hyperparameter optimization, including hyperparameter optimization, Bayesian optimization, and Gaussian processes. We also review BoTorch, GPyTorch and Ax, the new open-source frameworks that we use for Bayesian optimization, Gaussian process inference and adaptive experimentation, respectively. For experimentation, we apply Bayesian hyperparameter optimization, for optimizing group weights, to weighted group pooling, which couples unsupervised tiered graph autoencoders learning and supervised graph prediction learning for molecular graphs. We find that Ax, BoTorch and GPyTorch together provide a simple-to-use but powerful framework for Bayesian hyperparameter optimization, using Ax's high-level API that constructs and runs a full optimization loop and returns the best hyperparameter configuration.