# Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

arXiv: https://arxiv.org/abs/2201.06834 · HTML: https://mlsanity.com/paper/2201.06834/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 82 references that are in the map.
- First posted on arXiv 18 Jan 2022.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Neural Architecture Search with Bayesian Optimisation and Optimal Transport](https://mlsanity.com/paper/1802.07191/) (2018)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Multi-fidelity Bayesian Optimisation with Continuous Approximations](https://mlsanity.com/paper/1703.06240/) (2017)
- [Batch Bayesian Optimization via Local Penalization](https://mlsanity.com/paper/1505.08052/) (2015)
- [Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning](https://mlsanity.com/paper/1903.04703/) (2019)
- [Multi-Information Source Optimization](https://mlsanity.com/paper/1603.00389/) (2016)
- [HoloClean: Holistic Data Repairs with Probabilistic Inference](https://mlsanity.com/paper/1702.00820/) (2017)
- [Automated Machine Learning: From Principles to Practices](https://mlsanity.com/paper/1810.13306/) (2018)
- [OpenBox: A Generalized Black-box Optimization Service](https://mlsanity.com/paper/2106.00421/) (2021)
- [Rafiki: Machine Learning as an Analytics Service System](https://mlsanity.com/paper/1804.06087/) (2018)
- [Efficient Automatic CASH via Rising Bandits](https://mlsanity.com/paper/2012.04371/) (2020)
- [ZeroER: Entity Resolution using Zero Labeled Examples](https://mlsanity.com/paper/1908.06049/) (2019)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)

## Built on by (most cited first, 13 of 13)

- [Transfer Learning for Bayesian Optimization: A Survey](https://mlsanity.com/paper/2302.05927/) (2023)
- [Transfer Learning based Search Space Design for Hyperparameter Tuning](https://mlsanity.com/paper/2206.02511/) (2022)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning](https://mlsanity.com/paper/2206.02663/) (2022)
- [Multi-Fidelity Methods for Optimization: A Survey](https://mlsanity.com/paper/2402.09638/) (2024)
- [ASP: Automatic Selection of Proxy dataset for efficient AutoML](https://mlsanity.com/paper/2310.11478/) (2023)
- [Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Wait](https://mlsanity.com/paper/2305.17595/) (2023)
- [Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks](https://mlsanity.com/paper/2403.01888/) (2024)
- [FastBO: Fast HPO and NAS with Adaptive Fidelity Identification](https://mlsanity.com/paper/2409.00584/) (2024)
- [Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL](https://mlsanity.com/paper/2511.03695/) (2025)
- [Efficient End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2206.09423/) (2022)
- [Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble](https://mlsanity.com/paper/2402.01379/) (2024)
- [Bootstrap Sampling Rate Greater than 1.0 May Improve Random Forest Performance](https://mlsanity.com/paper/2410.04297/) (2024)

## Abstract (arXiv)

The ever-growing demand and complexity of machine learning are putting pressure on hyper-parameter tuning systems: while the evaluation cost of models continues to increase, the scalability of state-of-the-arts starts to become a crucial bottleneck. In this paper, inspired by our experience when deploying hyper-parameter tuning in a real-world application in production and the limitations of existing systems, we propose Hyper-Tune, an efficient and robust distributed hyper-parameter tuning framework. Compared with existing systems, Hyper-Tune highlights multiple system optimizations, including (1) automatic resource allocation, (2) asynchronous scheduling, and (3) multi-fidelity optimizer. We conduct extensive evaluations on benchmark datasets and a large-scale real-world dataset in production. Empirically, with the aid of these optimizations, Hyper-Tune outperforms competitive hyper-parameter tuning systems on a wide range of scenarios, including XGBoost, CNN, RNN, and some architectural hyper-parameters for neural networks. Compared with the state-of-the-art BOHB and A-BOHB, Hyper-Tune achieves up to 11.2x and 5.1x speedups, respectively.