mlsanityData to 14 Oct 2026

Cost-Efficient Online Hyperparameter Optimization

arXiv 2101.06590 ↗First posted 17 Jan 2021cs.LG cs.AI cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent work on hyperparameters optimization (HPO) has shown the possibility of training certain hyperparameters together with regular parameters. However, these online HPO algorithms still require running evaluation on a set of validation examples at each training step, steeply increasing the training cost. To decide when to query the validation loss, we model online HPO as a time-varying Bayesian optimization problem, on top of which we propose a novel textitcostly feedback setting to capture the concept of the query cost. Under this setting, standard algorithms are cost-inefficient as they evaluate on the validation set at every round. In contrast, the cost-efficient GP-UCB algorithm proposed in this paper queries the unknown function only when the model is less confident about current decisions. We evaluate our proposed algorithm by tuning hyperparameters online for VGG and ResNet on CIFAR-10 and ImageNet100. Our proposed online HPO algorithm reaches human expert-level performance within a single run of the experiment, while incurring only modest computational overhead compared to regular training.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 17 Jan 2021.

Builds on

20 of 42 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Contrastive Multiview Codingcited by 1,301
  4. 2016HyperNetworkscited by 1,027
  5. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  6. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  7. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  8. 2016Hyperparameter optimization with approximate gradientcited by 233
  9. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  10. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 132
  11. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
  12. 2014Freeze-Thaw Bayesian Optimizationcited by 90
8 more
  1. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 73
  2. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  3. 2018Adversarially Robust Optimization with Gaussian Processescited by 62
  4. 2016Time-Varying Gaussian Process Bandit Optimizationcited by 50
  5. 2020Implicit differentiation of Lasso-type models for hyperparameter optimizationcited by 34
  6. 2019BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimizationcited by 30
  7. 2019Bayesian Optimization for Iterative Learningcited by 19
  8. 2020Time-varying Gaussian Process Bandit Optimization with Non-constant Evaluation Timecited by 3

Built on by

Cited by year

Citing papers per year: 2021: 2.

2 papers.

  1. 2021Syndicated Bandits: A Framework for Auto Tuning Hyper-parameters in Contextual Bandit Algorithmscited by 10
  2. 2021Genealogical Population-Based Training for Hyperparameter Optimizationcited by 0