mlsanityData to 7 Oct 2026

On Implicit Bias in Overparameterized Bilevel Optimization

arXiv 2212.14032 ↗First posted 28 Dec 2022cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Many problems in machine learning involve bilevel optimization (BLO), including hyperparameter optimization, meta-learning, and dataset distillation. Bilevel problems consist of two nested sub-problems, called the outer and inner problems, respectively. In practice, often at least one of these sub-problems is overparameterized. In this case, there are many ways to choose among optima that achieve equivalent objective values. Inspired by recent studies of the implicit bias induced by optimization algorithms in single-level optimization, we investigate the implicit bias of gradient-based algorithms for bilevel optimization. We delineate two standard BLO methods -- cold-start and warm-start -- and show that the converged solution or long-run behavior depends to a large degree on these and other algorithmic choices, such as the hypergradient approximation. We also show that the inner solutions obtained by warm-start BLO can encode a surprising amount of information about the outer objective, even when the outer parameters are low-dimensional. We believe that implicit bias deserves as central a role in the study of bilevel optimization as it has attained in the study of single-level neural net optimization.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 61 of its 102 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

61 of 102 references.

References · 61

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  9. 2018On the Spectral Bias of Neural Networkscited by 1,321
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  12. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
38 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Meta-Learning with Implicit Gradientscited by 523
  6. 2020Dataset Condensation with Gradient Matchingcited by 510
  7. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  8. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  9. 2017Population Based Training of Neural Networkscited by 351
  10. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2016Hyperparameter optimization with approximate gradientcited by 231
  13. 2018Approximation Methods for Bilevel Programmingcited by 231
  14. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  15. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
  16. 2021Efficient and Modular Implicit Differentiationcited by 199
  17. 2018Truncated Back-propagation for Bilevel Optimizationcited by 187
  18. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  19. 2019Population Based Augmentation: Efficient Learning of Augmentation Policy Schedulescited by 153
  20. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  21. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  22. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  23. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 126
  24. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  25. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 104
  26. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  27. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  28. 2019Faster AutoAugment: Learning Augmentation Strategies using Backpropagationcited by 96
  29. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 72
  30. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  31. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  32. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  33. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  34. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  35. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  36. 2016Connecting Generative Adversarial Networks and Actor-Critic Methodscited by 49
  37. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  38. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 8; 2025: 5; 2026: 2.

30 papers.

Sort:

By area:

  1. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  2. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  3. 2023Data Distillation: A Surveycited by 86
  4. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  5. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  6. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  7. 2023Vision-Language Dataset Distillationcited by 18
  8. 2024Importance-Aware Adaptive Dataset Distillationcited by 18
  9. 2024Let Go of Your Labels with Unsupervised Transfercited by 14
  10. 2024What is Dataset Distillation Learning?cited by 13
  11. 2023Auxiliary Learning as an Asymmetric Bargaining Gamecited by 12
  12. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
18 more
  1. 2023Nystrom Method for Accurate and Scalable Implicit Differentiationcited by 8
  2. 2023Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Singlecited by 7
  3. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  4. 2024Color-Oriented Redundancy Reduction in Dataset Distillationcited by 6
  5. 2023A Generalized Alternating Method for Bilevel Learning under the Polyak-Lojasiewicz Conditioncited by 5
  6. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  7. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 5
  8. 2025A First-order Generative Bilevel Optimization Framework for Diffusion Modelscited by 4
  9. 2026Motion Attribution for Video Generationcited by 4
  10. 2023CMVAE: Causal Meta VAE for Unsupervised Meta-Learningcited by 3
  11. 2023Test like you Train in Implicit Deep Learningcited by 3
  12. 2024A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learningcited by 3
  13. 2025Bi-Level Policy Optimization with Nystr"om Hypergradientscited by 3
  14. 2023Gradient constrained sharpness-aware prompt learning for vision-language modelscited by 1
  15. 2025Accelerating Optimization via Differentiable Stopping Timecited by 1
  16. 2023A Challenge in Reweighting Data with Bilevel Optimizationcited by 0
  17. 2024Multi Task Inverse Reinforcement Learning for Common Sense Rewardcited by 0
  18. 2026Bilevel Graph Structure Learning, Revisited: Inner-Channel Origins of the Reported Gaincited by 0