mlsanityData to 14 Oct 2026

Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization

arXiv 2406.14095 ↗First posted 20 Jun 2024cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Bi-level optimization (BO) has become a fundamental mathematical framework for addressing hierarchical machine learning problems. As deep learning models continue to grow in size, the demand for scalable bi-level optimization solutions has become increasingly critical. Traditional gradient-based bi-level optimization algorithms, due to their inherent characteristics, are ill-suited to meet the demands of large-scale applications. In this paper, we introduce Forward Gradient Unrolling with Forward Fradient, abbreviated as (FG)2U, which achieves an unbiased stochastic approximation of the meta gradient for bi-level optimization. (FG)2U circumvents the memory and approximation issues associated with classical bi-level optimization approaches, and delivers significantly more accurate gradient estimates than existing large-scale bi-level optimization approaches. Additionally, (FG)2U is inherently designed to support parallel computing, enabling it to effectively leverage large-scale distributed computing systems to achieve significant computational efficiency. In practice, (FG)2U and other methods can be strategically placed at different stages of the training process to achieve a more cost-effective two-phase paradigm. Further, (FG)2U is easy to implement within popular deep learning frameworks, and can be conveniently adapted to address more challenging zeroth-order bi-level optimization scenarios. We provide a thorough convergence analysis and a comprehensive practical discussion for (FG)2U, complemented by extensive empirical evaluations, showcasing its superior performance in diverse large-scale bi-level optimization tasks. Code is available at https://github.com/ShenQianli/FG2U.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 53 of its 76 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

53 of 76 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  11. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  12. 2016Learning to learn by gradient descent by gradient descentcited by 620
38 more
  1. 2019Meta-Learning with Implicit Gradientscited by 525
  2. 2020Dataset Condensation with Gradient Matchingcited by 516
  3. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  5. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  6. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  8. 2021Physics-informed neural networks with hard constraints for inverse designcited by 280
  9. 2018Approximation Methods for Bilevel Programmingcited by 236
  10. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  12. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  13. 2018Truncated Back-propagation for Bilevel Optimizationcited by 190
  14. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  15. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  16. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  17. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 129
  18. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  19. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  20. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 120
  21. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 120
  22. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  23. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  24. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  25. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  26. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  27. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 79
  28. 2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 76
  29. 2022Gradients without Backpropagationcited by 73
  30. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  31. 2022Scaling Forward Gradient With Local Lossescited by 65
  32. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  33. 2022Advancing Model Pruning via Bi-level Optimizationcited by 56
  34. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  35. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  36. 2022Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradientscited by 17
  37. 2023Making Scalable Meta Learning Practicalcited by 17
  38. 2020Zeroth-order Deterministic Policy Gradientcited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 2.

11 papers.

  1. 2024EraseAnything: Enabling Concept Erasure in Rectified Flow Transformerscited by 95
  2. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  3. 2025Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbationscited by 11
  4. 2025Scalable Meta-Learning via Mixed-Mode Differentiationcited by 4
  5. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  6. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradientscited by 1
  7. 2025Bridging Constraints and Stochasticity: A Fully First-Order Method for Stochastic Bilevel Optimization with Linear Constraintscited by 1
  8. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  9. 2025End-to-End Dialog Neural Coreference Resolution: Balancing Efficiency and Accuracy in Large-Scale Systemscited by 0
  10. 2025Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraintscited by 0
  11. 2026Bilevel Optimization of Topology and Hyperparameters (BOTH)cited by 0