mlsanityData to 7 Oct 2026

Stability and Generalization of Bilevel Programming in Hyperparameter Optimization

arXiv 2106.04188 ↗First posted 8 Jun 2021cs.LG math.OCcited by 31

Not on the map. Why?

Abstract (arXiv)

The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r.t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the classical cross-validation algorithm. Our results suggest that gradient-based algorithms can be better than cross-validation under certain conditions in a theoretical perspective. Furthermore, we prove that regularization terms in both the outer and inner levels can relieve the overfitting problem in gradient-based algorithms. In experiments on feature learning and data reweighting for noisy labels, we corroborate our theoretical findings.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 23 of its 54 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

23 of 54 references.

References · 23

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  5. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  6. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  7. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  8. 2019Residual Flows for Invertible Generative Modelingcited by 256
  9. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  10. 2016Hyperparameter optimization with approximate gradientcited by 231
  11. 2018Approximation Methods for Bilevel Programmingcited by 231
  12. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
11 more
  1. 2018Truncated Back-propagation for Bilevel Optimizationcited by 187
  2. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  3. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 104
  4. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 72
  5. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  6. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  7. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  8. 2020Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameterscited by 65
  9. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  10. 2020On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networkscited by 19
  11. 2016DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 4; 2024: 8; 2025: 6; 2026: 8.

31 papers.

Sort:

By area:

  1. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 97
  2. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 50
  3. 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 30
  4. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  5. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  6. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  7. 2022Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradientscited by 17
  8. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  9. 2024On the Complexity of First-Order Methods in Stochastic Bilevel Optimizationcited by 11
  10. 2023Non-Convex Bilevel Optimization with Time-Varying Objective Functionscited by 8
  11. 2024A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimizationcited by 8
  12. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
19 more
  1. 2025A Fully First-Order Layer for Differentiable Optimizationcited by 3
  2. 2026Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimizationcited by 3
  3. 2025Learning Theory for Kernel Bilevel Optimizationcited by 2
  4. 2025Few-Shot Learning in Video and 3D Object Detection: A Surveycited by 2
  5. 2025On the Condition Number Dependency in Bilevel Optimizationcited by 2
  6. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  7. 2022On Stability and Generalization of Bilevel Optimization Problemcited by 1
  8. 2024Exploring the Generalization Capabilities of AID-based Bi-level Optimizationcited by 1
  9. 2025Faster Gradient Methods for Highly-Smooth Stochastic Bilevel Optimizationcited by 1
  10. 2026Stochastic variance reduced extragradient methods for solving hierarchical variational inequalitiescited by 1
  11. 2022A Globally Convergent Gradient-based Bilevel Hyperparameter Optimization Methodcited by 0
  12. 2023Decentralized and Privacy-Preserving Learning of Approximate Stackelberg Solutions in Energy Trading Games with Demand Response Aggregatorscited by 0
  13. 2024Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimizationcited by 0
  14. 2024Riemannian Bilevel Optimizationcited by 0
  15. 2026On the Convergence of Single-Loop Stochastic Bilevel Optimization with Approximate Implicit Differentiationcited by 0
  16. 2026Variational Rectification Inference for Learning with Noisy Labelscited by 0
  17. 2026Meta Additive Model: Interpretable Sparse Learning With Auto Weightingcited by 0
  18. 2026On the Stability and Generalization of First-order Bilevel Minimax Optimizationcited by 0
  19. 2026Sharper Analysis of Single-Loop Methods for Bilevel Optimizationcited by 0