# Genealogical Population-Based Training for Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2109.14925 · HTML: https://mlsanity.com/paper/2109.14925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits](https://mlsanity.com/paper/2002.02518/) (2020)
- [Time-Varying Gaussian Process Bandit Optimization](https://mlsanity.com/paper/1601.06650/) (2016)
- [Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures](https://mlsanity.com/paper/1608.06037/) (2016)
- [Bayesian Optimization for Iterative Learning](https://mlsanity.com/paper/1909.09593/) (2019)
- [Faster Improvement Rate Population Based Training](https://mlsanity.com/paper/2109.13800/) (2021)
- [Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1807.07362/) (2018)
- [Cost-Efficient Online Hyperparameter Optimization](https://mlsanity.com/paper/2101.06590/) (2021)
- [Hyper-parameter optimization of Deep Convolutional Networks for object recognition](https://mlsanity.com/paper/1501.07645/) (2015)

## Abstract (arXiv)

HyperParameter Optimization (HPO) aims at finding the best HyperParameters (HPs) of learning models, such as neural networks, in the fastest and most efficient way possible. Most recent HPO algorithms try to optimize HPs regardless of the model that obtained them, assuming that for different models, same HPs will produce very similar results. We break free from this paradigm and propose a new take on preexisting methods that we called Genealogical Population Based Training (GPBT). GPBT, via the shared histories of "genealogically"-related models, exploit the coupling of HPs and models in an efficient way. We experimentally demonstrate that our method cuts down by 2 to 3 times the computational cost required, generally allows a 1% accuracy improvement on computer vision tasks, and reduces the variance of the results by an order of magnitude, compared to the current algorithms. Our method is search-algorithm agnostic so that the inner search routine can be any search algorithm like TPE, GP, CMA or random search.