# HyperJump: Accelerating HyperBand via Risk Modelling

arXiv: https://arxiv.org/abs/2108.02479 · HTML: https://mlsanity.com/paper/2108.02479/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size](https://mlsanity.com/paper/2009.00437/) (2020)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Multi-Information Source Optimization](https://mlsanity.com/paper/1603.00389/) (2016)
- [A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1811.00755/) (2018)
- [Staleness-aware Async-SGD for Distributed Deep Learning](https://mlsanity.com/paper/1511.05950/) (2015)
- [Why Non-myopic Bayesian Optimization is Promising and How Far Should We Look-ahead? A Study via Rollout](https://mlsanity.com/paper/1911.01004/) (2019)
- [Multi-fidelity Bayesian Optimization with Max-value Entropy Search and its parallelization](https://mlsanity.com/paper/1901.08275/) (2019)
- [Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning](https://mlsanity.com/paper/1801.01596/) (2018)
- [TrimTuner: Efficient Optimization of Machine Learning Jobs in the Cloud via Sub-Sampling](https://mlsanity.com/paper/2011.04726/) (2020)
- [Lynceus: Cost-efficient Tuning and Provisioning of Data Analytic Jobs](https://mlsanity.com/paper/1905.02119/) (2019)
- [Exact Inference for Gaussian Process Regression in case of Big Data with the Cartesian Product Structure](https://mlsanity.com/paper/1403.6573/) (2014)

## Built on by (most cited first, 4 of 4)

- [Supervising the Multi-Fidelity Race of Hyperparameter Configurations](https://mlsanity.com/paper/2202.09774/) (2022)
- [Hyper-parameter Tuning for Adversarially Robust Models](https://mlsanity.com/paper/2304.02497/) (2023)
- [Iterative Deepening Hyperband](https://mlsanity.com/paper/2302.00511/) (2023)
- [POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence](https://mlsanity.com/paper/2505.11745/) (2025)

## Abstract (arXiv)

In the literature on hyper-parameter tuning, a number of recent solutions rely on low-fidelity observations (e.g., training with sub-sampled datasets) in order to efficiently identify promising configurations to be then tested via high-fidelity observations (e.g., using the full dataset). Among these, HyperBand is arguably one of the most popular solutions, due to its efficiency and theoretically provable robustness. In this work, we introduce HyperJump, a new approach that builds on HyperBand's robust search strategy and complements it with novel model-based risk analysis techniques that accelerate the search by skipping the evaluation of low risk configurations, i.e., configurations that are likely to be eventually discarded by HyperBand. We evaluate HyperJump on a suite of hyper-parameter optimization problems and show that it provides over one-order of magnitude speed-ups, both in sequential and parallel deployments, on a variety of deep-learning, kernel-based learning, and neural architectural search problems when compared to HyperBand and to several state-of-the-art optimizers.