# HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2110.01698 · HTML: https://mlsanity.com/paper/2110.01698/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Survey of Dropout Methods for Deep Neural Networks](https://mlsanity.com/paper/1904.13310/) (2019)
- [Radon Inversion via Deep Learning](https://mlsanity.com/paper/1808.03015/) (2018)
- [Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations](https://mlsanity.com/paper/2010.00072/) (2020)
- [Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm](https://mlsanity.com/paper/2006.12703/) (2020)
- [Illumination Pattern Design with Deep Learning for Single-Shot Fourier Ptychographic Microscopy](https://mlsanity.com/paper/1810.03481/) (2018)
- [Surrogate Optimization of Deep Neural Networks for Groundwater Predictions](https://mlsanity.com/paper/1908.10947/) (2019)
- [A hierarchical approach to deep learning and its application to tomographic reconstruction](https://mlsanity.com/paper/1912.07743/) (2019)
- [Deep Learning Based Computed Tomography Whys and Wherefores](https://mlsanity.com/paper/1904.03908/) (2019)
- [Optimal Physical Preprocessing for Example-Based Super-Resolution](https://mlsanity.com/paper/1807.04813/) (2018)

## Built on by (most cited first, 1 of 1)

- [Hyperparameter Optimization of Generative Adversarial Network Models for High-Energy Physics Simulations](https://mlsanity.com/paper/2208.07715/) (2022)

## Abstract (arXiv)

We present a new software, HYPPO, that enables the automatic tuning of hyperparameters of various deep learning (DL) models. Unlike other hyperparameter optimization (HPO) methods, HYPPO uses adaptive surrogate models and directly accounts for uncertainty in model predictions to find accurate and reliable models that make robust predictions. Using asynchronous nested parallelism, we are able to significantly alleviate the computational burden of training complex architectures and quantifying the uncertainty. HYPPO is implemented in Python and can be used with both TensorFlow and PyTorch libraries. We demonstrate various software features on time-series prediction and image classification problems as well as a scientific application in computed tomography image reconstruction. Finally, we show that (1) we can reduce by an order of magnitude the number of evaluations necessary to find the most optimal region in the hyperparameter space and (2) we can reduce by two orders of magnitude the throughput for such HPO process to complete.