# FlexHB: a More Efficient and Flexible Framework for Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2402.13641 · HTML: https://mlsanity.com/paper/2402.13641/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Multi-fidelity Bayesian Optimisation with Continuous Approximations](https://mlsanity.com/paper/1703.06240/) (2017)
- [Maximizing acquisition functions for Bayesian optimization](https://mlsanity.com/paper/1805.10196/) (2018)
- [Batch Bayesian Optimization via Local Penalization](https://mlsanity.com/paper/1505.08052/) (2015)
- [Multi-Information Source Optimization](https://mlsanity.com/paper/1603.00389/) (2016)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions](https://mlsanity.com/paper/1410.7827/) (2014)
- [Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly](https://mlsanity.com/paper/1903.06694/) (2019)
- [Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL](https://mlsanity.com/paper/2006.13799/) (2020)
- [MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements](https://mlsanity.com/paper/2012.03011/) (2020)
- [Accelerating Neural Architecture Search using Performance Prediction](https://mlsanity.com/paper/1705.10823/) (2017)
- [Noisy Blackbox Optimization with Multi-Fidelity Queries: A Tree Search Approach](https://mlsanity.com/paper/1810.10482/) (2018)
- [An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization](https://mlsanity.com/paper/2007.05670/) (2020)

## Abstract (arXiv)

Given a Hyperparameter Optimization(HPO) problem, how to design an algorithm to find optimal configurations efficiently? Bayesian Optimization(BO) and the multi-fidelity BO methods employ surrogate models to sample configurations based on history evaluations. More recent studies obtain better performance by integrating BO with HyperBand(HB), which accelerates evaluation by early stopping mechanism. However, these methods ignore the advantage of a suitable evaluation scheme over the default HyperBand, and the capability of BO is still constrained by skewed evaluation results. In this paper, we propose FlexHB, a new method pushing multi-fidelity BO to the limit as well as re-designing a framework for early stopping with Successive Halving(SH). Comprehensive study on FlexHB shows that (1) our fine-grained fidelity method considerably enhances the efficiency of searching optimal configurations, (2) our FlexBand framework (self-adaptive allocation of SH brackets, and global ranking of configurations in both current and past SH procedures) grants the algorithm with more flexibility and improves the anytime performance. Our method achieves superior efficiency and outperforms other methods on various HPO tasks. Empirical results demonstrate that FlexHB can achieve up to 6.9X and 11.1X speedups over the state-of-the-art MFES-HB and BOHB respectively.