mlsanityData to 7 Oct 2026

Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems

arXiv 2206.05976 ↗First posted 13 Jun 2022math.OC cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Gradient-based optimization methods for hyperparameter tuning guarantee theoretical convergence to stationary solutions when for fixed upper-level variable values, the lower level of the bilevel program is strongly convex (LLSC) and smooth (LLS). This condition is not satisfied for bilevel programs arising from tuning hyperparameters in many machine learning algorithms. In this work, we develop a sequentially convergent Value Function based Difference-of-Convex Algorithm with inexactness (VF-iDCA). We show that this algorithm achieves stationary solutions without LLSC and LLS assumptions for bilevel programs from a broad class of hyperparameter tuning applications. Our extensive experiments confirm our theoretical findings and show that the proposed VF-iDCA yields superior performance when applied to tune hyperparameters.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

22 of 45 references.

References · 22

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2019Meta-Learning with Implicit Gradientscited by 523
  4. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  5. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  6. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  7. 2016Hyperparameter optimization with approximate gradientcited by 231
  8. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
  9. 2018Truncated Back-propagation for Bilevel Optimizationcited by 187
  10. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  11. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 78
  12. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 72
10 more
  1. 2020Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameterscited by 65
  2. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 62
  3. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  4. 2022Will Bilevel Optimizers Benefit from Loopscited by 36
  5. 2020Implicit differentiation of Lasso-type models for hyperparameter optimizationcited by 34
  6. 2021A General Descent Aggregation Framework for Gradient-based Bi-level Optimizationcited by 28
  7. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  8. 2021Difference of convex algorithms for bilevel programs with applications in hyperparameter selectioncited by 17
  9. 2017Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functionscited by 3
  10. 2018A refined convergence analysis of pDCAe with applications to simultaneous sparse recovery and outlier detectioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 4; 2025: 3; 2026: 1.

15 papers.

Sort:

  1. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  2. 2022A Conditional Gradient-based Method for Simple Bilevel Optimization with Convex Lower-level Problemcited by 17
  3. 2023Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oraclescited by 16
  4. 2024Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithmcited by 16
  5. 2024Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategycited by 15
  6. 2022Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimizationcited by 10
  7. 2023Moreau Envelope Based Difference-of-weakly-Convex Reformulation and Algorithm for Bilevel Programscited by 7
  8. 2024A Primal-Dual-Assisted Penalty Approach to Bilevel Optimization with Coupled Constraintscited by 7
  9. 2023A Generalized Alternating Method for Bilevel Learning under the Polyak-Lojasiewicz Conditioncited by 5
  10. 2023Multi-task Meta Label Correction for Time Series Predictioncited by 1
  11. 2026Penalty-Based First-Order Methods for Bilevel Optimization with Minimax and Constrained Lower-Level Problemscited by 1
  12. 2024Hyperparameter Tuning Through Pessimistic Bilevel Optimizationcited by 0
3 more
  1. 2025Sequential QCQP for Bilevel Optimization with Line Searchcited by 0
  2. 2025A Single-Loop First-Order Algorithm for Linearly Constrained Bilevel Optimizationcited by 0
  3. 2025Bilevel Models for Adversarial Learning and A Case Studycited by 0