mlsanityData to 13 Oct 2026

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

arXiv 2209.08709 ↗First posted 19 Sep 2022cs.LG cs.AI math.OCcited by 103

Not on the map. Why?

Abstract (arXiv)

Bilevel optimization (BO) is useful for solving a variety of important machine learning problems including but not limited to hyperparameter optimization, meta-learning, continual learning, and reinforcement learning. Conventional BO methods need to differentiate through the low-level optimization process with implicit differentiation, which requires expensive calculations related to the Hessian matrix. There has been a recent quest for first-order methods for BO, but the methods proposed to date tend to be complicated and impractical for large-scale deep learning applications. In this work, we propose a simple first-order BO algorithm that depends only on first-order gradient information, requires no implicit differentiation, and is practical and efficient for large-scale non-convex functions in deep learning. We provide non-asymptotic convergence analysis of the proposed method to stationary points for non-convex objectives and present empirical results that show its superior practical performance.

Key facts. Cited by 103 papers in the map (data through 13 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

32 of 55 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,328
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,882
  4. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 529
  5. 2019Meta-Learning with Implicit Gradientscited by 523
  6. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 514
  7. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 404
  8. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  9. 2018Approximation Methods for Bilevel Programmingcited by 234
  10. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 233
  11. 2016Hyperparameter optimization with approximate gradientcited by 232
  12. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
20 more
  1. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
  2. 2017Training GANs with Optimismcited by 217
  3. 2018Truncated Back-propagation for Bilevel Optimizationcited by 188
  4. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  5. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  6. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  7. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  8. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
  9. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 79
  10. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  11. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  12. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
  13. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  14. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  15. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  16. 2019Penalty Method for Inversion-Free Deep Bilevel Optimizationcited by 33
  17. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  18. 2021Value-Function-based Sequential Minimization for Bi-level Optimizationcited by 24
  19. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  20. 2018Learning to Defend by Learning to Attackcited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 21; 2024: 36; 2025: 27; 2026: 18.

103 papers.

Sort:

By area:

  1. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 97
  2. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  3. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  4. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  5. 2024Erasing Undesirable Influence in Diffusion Modelscited by 56
  6. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 51
  7. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  8. 2023First-order penalty methods for bilevel optimizationcited by 40
  9. 2023On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysiscited by 39
  10. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  11. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  12. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
38 more
  1. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 30
  2. 2023Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexitycited by 26
  3. 2023Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Informationcited by 18
  4. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  5. 2023Making Scalable Meta Learning Practicalcited by 17
  6. 2023A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimizationcited by 16
  7. 2023Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oraclescited by 16
  8. 2024Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithmcited by 16
  9. 2024Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategycited by 16
  10. 2024Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomycited by 16
  11. 2024Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Modelscited by 16
  12. 2024Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysiscited by 15
  13. 2023On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Levelcited by 14
  14. 2023SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learningcited by 14
  15. 2024Let Go of Your Labels with Unsupervised Transfercited by 14
  16. 2025Cautious Weight Decaycited by 13
  17. 2024Game-Theoretic Unlearnable Example Generatorcited by 12
  18. 2024On the Complexity of First-Order Methods in Stochastic Bilevel Optimizationcited by 12
  19. 2024Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimizationcited by 11
  20. 2024First-Order Methods for Linearly Constrained Bilevel Optimizationcited by 10
  21. 2024Optimal Hessian/Jacobian-Free Nonconvex-PL Bilevel Optimizationcited by 10
  22. 2024An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 9
  23. 2024A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimizationcited by 8
  24. 2024BiLO: Bilevel Local Operator Learning for PDE Inverse Problemscited by 8
  25. 2025Direct Prediction Set Minimization via Bilevel Conformal Classifier Trainingcited by 8
  26. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  27. 2024A Primal-Dual-Assisted Penalty Approach to Bilevel Optimization with Coupled Constraintscited by 7
  28. 2024A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 7
  29. 2024Fully Zeroth-Order Bilevel Programming via Gaussian Smoothingcited by 6
  30. 2023A Generalized Alternating Method for Bilevel Learning under the Polyak-Lojasiewicz Conditioncited by 5
  31. 2024Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysiscited by 5
  32. 2025Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Trainingcited by 5
  33. 2025A Survey on Unlearnable Datacited by 5
  34. 2023An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learningcited by 4
  35. 2024LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspacecited by 4
  36. 2025A First-order Generative Bilevel Optimization Framework for Diffusion Modelscited by 4
  37. 2025Stochastic Bilevel Optimization with Heavy-Tailed Noisecited by 4
  38. 2025Trellis: Learning to Compress Key-Value Memory in Attention Modelscited by 4

50 most cited of 103 All 103 (JSON)