mlsanityData to 14 Oct 2026

Bilevel Programming for Hyperparameter Optimization and Meta-Learning

arXiv 1806.04910 ↗First posted 13 Jun 2018stat.ML cs.LGcited by 405

Not on the map. Why?

Abstract (arXiv)

We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.

Key facts. Cited by 405 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 66; 2022: 79; 2023: 83; 2024: 62; 2025: 48; 2026: 67.

405 papers.

Sort:

By area:

  1. 2021Efficient and Modular Implicit Differentiationcited by 203
  2. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  3. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
  4. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  5. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  6. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  7. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  8. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  9. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  10. 2021Implicit MLE: Backpropagating Through Discrete Exponential Family Distributionscited by 98
  11. 2024EraseAnything: Enabling Concept Erasure in Rectified Flow Transformerscited by 95
  12. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 94
38 more
  1. 2023Data Distillation: A Surveycited by 87
  2. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  3. 2021HyperMorph: Amortized Hyperparameter Learning for Image Registrationcited by 85
  4. 2022Tutorial on amortized optimizationcited by 83
  5. 2022When Do Flat Minima Optimizers Work?cited by 78
  6. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  7. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  8. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  9. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  10. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  11. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
  12. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  13. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  14. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  15. 2021An End-to-End Framework for Molecular Conformation Generation via Bilevel Programmingcited by 59
  16. 2023A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyondcited by 55
  17. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  18. 2022Meta Balanced Network for Fair Face Recognitioncited by 53
  19. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  20. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  21. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 51
  22. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  23. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  24. 2023First-order penalty methods for bilevel optimizationcited by 41
  25. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 41
  26. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  27. 2021Multi-Objective Meta Learningcited by 39
  28. 2023On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysiscited by 39
  29. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  30. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  31. 2022Contrastive Meta Learning with Behavior Multiplicity for Recommendationcited by 37
  32. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 37
  33. 2021Contrastive Learning Improves Model Robustness Under Label Noisecited by 36
  34. 2022Will Bilevel Optimizers Benefit from Loopscited by 36
  35. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  36. 2023Online Hyperparameter Optimization for Class-Incremental Learningcited by 36
  37. 2021On the Convergence Theory for Hessian-Free Bilevel Algorithmscited by 35
  38. 2022Learning the Effect of Registration Hyperparameters with HyperMorphcited by 35

50 most cited of 405 All 405 (JSON)