mlsanityData to 7 Oct 2026

Supervising the Multi-Fidelity Race of Hyperparameter Configurations

arXiv 2202.09774 ↗First posted 20 Feb 2022cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 27 of its 54 references that are in the map. First posted on arXiv 20 Feb 2022.

Builds on

27 of 54 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  4. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  5. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  6. 2015Deep Kernel Learningcited by 440
  7. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  8. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  9. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  10. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  11. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  12. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
15 more
  1. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  2. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  3. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 120
  4. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  5. 2016Multi-Information Source Optimizationcited by 83
  6. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  7. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  8. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  9. 2020Multi-Fidelity Bayesian Optimization via Deep Neural Networkscited by 46
  10. 2020Using a thousand optimization tasks to learn hyperparameter search strategiescited by 29
  11. 2020Learning to Rank Learning Curvescited by 16
  12. 2018Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learningcited by 12
  13. 2019Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settingscited by 12
  14. 2020TrimTuner: Efficient Optimization of Machine Learning Jobs in the Cloud via Sub-Samplingcited by 12
  15. 2021HyperJump: Accelerating HyperBand via Risk Modellingcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 6; 2025: 5.

15 papers.

Sort:

  1. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  2. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 17
  3. 2023Scaling Laws for Hyperparameter Optimizationcited by 16
  4. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  5. 2024Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structurecited by 9
  6. 2024Multi-Fidelity Methods for Optimization: A Surveycited by 5
  7. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  8. 2023Deep Pipeline Embeddings for AutoMLcited by 2
  9. 2024Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolationcited by 2
  10. 2025Tune My Adam, Please!cited by 2
  11. 2025Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuningcited by 2
  12. 2024FastBO: Fast HPO and NAS with Adaptive Fidelity Identificationcited by 1
3 more
  1. 2025Derivation of Output Correlation Inferences for Multi-Output (aka Multi-Task) Gaussian Processcited by 1
  2. 2025Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networkscited by 0
  3. 2025Quickly Tuning Foundation Models for Image Segmentationcited by 0