Exploring the Generalization Capabilities of AID-based Bi-level Optimization
Not on the map. Why?
Abstract (arXiv)
Bi-level optimization has achieved considerable success in contemporary machine learning applications, especially for given proper hyperparameters. However, due to the two-level optimization structure, commonly, researchers focus on two types of bi-level optimization methods: approximate implicit differentiation (AID)-based and iterative differentiation (ITD)-based approaches. ITD-based methods can be readily transformed into single-level optimization problems, facilitating the study of their generalization capabilities. In contrast, AID-based methods cannot be easily transformed similarly but must stay in the two-level structure, leaving their generalization properties enigmatic. In this paper, although the outer-level function is nonconvex, we ascertain the uniform stability of AID-based methods, which achieves similar results to a single-level nonconvex problem. We conduct a convergence analysis for a carefully chosen step size to maintain stability. Combining the convergence and stability results, we give the generalization ability of AID-based bi-level optimization methods. Furthermore, we carry out an ablation study of the parameters and assess the performance of these methods on real-world tasks. Our experimental results corroborate the theoretical findings, demonstrating the effectiveness and potential applications of these methods.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 25 Nov 2024.
Builds on
37 of 52 references.
References · 37
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
- 2019Meta-Learning with Implicit Gradientscited by 525
- 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
- 2018Meta-learning with differentiable closed-form solverscited by 397
- 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
- 2018Approximation Methods for Bilevel Programmingcited by 236
- 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
- 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
- 2020On the Iteration Complexity of Hypergradient Computationcited by 155
25 more
- 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
- 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
- 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
- 2020Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameterscited by 65
- 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
- 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
- 2022Advancing Model Pruning via Bi-level Optimizationcited by 56
- 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
- 2016Connecting Generative Adversarial Networks and Actor-Critic Methodscited by 50
- 2018Deep Bilevel Learningcited by 49
- 2022Topology-aware Generalization of Decentralized SGDcited by 38
- 2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
- 2022Will Bilevel Optimizers Benefit from Loopscited by 36
- 2020Implicit differentiation of Lasso-type models for hyperparameter optimizationcited by 34
- 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
- 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
- 2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
- 2021CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signalscited by 22
- 2020On the Global Optimality of Model-Agnostic Meta-Learningcited by 21
- 2022Decentralized Stochastic Bilevel Optimization with Improved per-Iteration Complexitycited by 21
- 2022What is a Good Metric to Study Generalization of Minimax Learners?cited by 16
- 2017Anti-Makeup: Learning A Bi-Level Adversarial Network for Makeup-Invariant Face Verificationcited by 9
- 2023Stability and Generalization of the Decentralized Stochastic Gradient Descent Ascent Algorithmcited by 9
- 2021Projection-Free Algorithm for Stochastic Bi-level Optimizationcited by 8
- 2019Automatic Design of CNNs via Differentiable Neural Architecture Search for PolSAR Image Classificationcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.