mlsanityData to 14 Oct 2026

Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

arXiv 2110.10461 ↗First posted 20 Oct 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Machine learning training methods depend plentifully and intricately on hyperparameters, motivating automated strategies for their optimisation. Many existing algorithms restart training for each new hyperparameter choice, at considerable computational cost. Some hypergradient-based one-pass methods exist, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or take several times longer to train than their base models. We extend these existing methods to develop an approximate hypergradient-based hyperparameter optimiser which is applicable to any continuous hyperparameter appearing in a differentiable model weight update, yet requires only one training episode, with no restarts. We also provide a motivating argument for convergence to the true hypergradient, and perform tractable gradient-based optimisation of independent learning rates for each model parameter. Our method performs competitively from varied random hyperparameter initialisations on several UCI datasets and Fashion-MNIST (using a one-layer MLP), Penn Treebank (using an LSTM) and CIFAR-10 (using a ResNet-18), in time only 2-3x greater than vanilla training.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

37 of 70 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  7. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  8. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  9. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  10. 2017Population Based Training of Neural Networkscited by 367
  11. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  12. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
25 more
  1. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  2. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  3. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  4. 2018A System for Massively Parallel Hyperparameter Tuningcited by 202
  5. 2018Truncated Back-propagation for Bilevel Optimizationcited by 190
  6. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 182
  7. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  8. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 121
  9. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  10. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
  11. 2019Generalized Inner Loop Meta-Learningcited by 93
  12. 2014Freeze-Thaw Bayesian Optimizationcited by 90
  13. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 79
  14. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 73
  15. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  16. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
  17. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  18. 2019Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learningcited by 50
  19. 2019Penalty Method for Inversion-Free Deep Bilevel Optimizationcited by 33
  20. 2020Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guaranteecited by 28
  21. 2014Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spacescited by 16
  22. 2015Selecting Near-Optimal Learners via Incremental Data Allocationcited by 12
  23. 2018Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learningcited by 12
  24. 2020Adaptive Hierarchical Hyper-gradient Descentcited by 2
  25. 2020Online Hyperparameter Search Interleaved with Proximal Parameter Updatescited by 1

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 4.

7 papers.

  1. 2022Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Predictioncited by 21
  2. 2022Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradientscited by 17
  3. 2023Making Scalable Meta Learning Practicalcited by 17
  4. 2022Simple and Effective Gradient-Based Tuning of Sequence-to-Sequence Modelscited by 2
  5. 2023Online Laplace Model Selection Revisitedcited by 2
  6. 2023Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lenscited by 2
  7. 2023Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networkscited by 1