mlsanityData to 7 Oct 2026

Hyperparameter optimization with approximate gradient

arXiv 1602.02355 ↗First posted 7 Feb 2016stat.ML cs.LG math.OCcited by 231

Not on the map. Why?

Abstract (arXiv)

Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 45; 2023: 46; 2024: 44; 2025: 30; 2026: 28.

231 papers.

Sort:

By area:

  1. 2021Graph Contrastive Learning Automatedcited by 287
  2. 2021Efficient and Modular Implicit Differentiationcited by 199
  3. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  4. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  5. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  6. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  7. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 103
  8. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  9. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  10. 2021HyperMorph: Amortized Hyperparameter Learning for Image Registrationcited by 84
  11. 2021On Training Implicit Modelscited by 73
  12. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
38 more
  1. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  2. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 62
  3. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 62
  4. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  5. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  6. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  7. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  8. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  9. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 50
  10. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  11. 2022Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learningcited by 44
  12. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  13. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  14. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  15. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  16. 2023First-order penalty methods for bilevel optimizationcited by 40
  17. 2021Multi-Objective Meta Learningcited by 39
  18. 2023On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysiscited by 39
  19. 2022Will Bilevel Optimizers Benefit from Loopscited by 36
  20. 2022Learning the Effect of Registration Hyperparameters with HyperMorphcited by 35
  21. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  22. 2021On the Convergence Theory for Hessian-Free Bilevel Algorithmscited by 33
  23. 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
  24. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
  25. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  26. 2021A General Descent Aggregation Framework for Gradient-based Bi-level Optimizationcited by 28
  27. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  28. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  29. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  30. 2023Probabilistic Bilevel Coreset Selectioncited by 26
  31. 2023Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexitycited by 26
  32. 2022Automatic differentiation of nonsmooth iterative algorithmscited by 25
  33. 2021Value-Function-based Sequential Minimization for Bi-level Optimizationcited by 24
  34. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  35. 2022Fair Representation Learning through Implicit Path Alignmentcited by 22
  36. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 22
  37. 2022Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Predictioncited by 21
  38. 2022Decentralized Stochastic Bilevel Optimization with Improved per-Iteration Complexitycited by 21

50 most cited of 231 All 231 (JSON)