mlsanityData to 7 Oct 2026

Betty: An Automatic Differentiation Library for Multilevel Optimization

arXiv 2207.02849 ↗First posted 5 Jul 2022cs.LG cs.AI math.OCcited by 27

Not on the map. Why?

Abstract (arXiv)

Gradient-based multilevel optimization (MLO) has gained attention as a framework for studying numerous problems, ranging from hyperparameter optimization and meta-learning to neural architecture search and reinforcement learning. However, gradients in MLO, which are obtained by composing best-response Jacobians via the chain rule, are notoriously difficult to implement and memory/compute intensive. We take an initial step towards closing this gap by introducing Betty, a software library for large-scale MLO. At its core, we devise a novel dataflow graph for MLO, which allows us to (1) develop efficient automatic differentiation for MLO that reduces the computational complexity from O(d^3) to O(d^2), (2) incorporate systems support such as mixed-precision and data-parallel training for scalability, and (3) facilitate implementation of MLO programs of arbitrary complexity while allowing a modular interface for diverse algorithmic and systems design choices. We empirically demonstrate that Betty can be used to implement an array of MLO programs, while also observing up to 11% increase in test accuracy, 14% decrease in GPU memory usage, and 20% decrease in training wall time over existing implementations on multiple benchmarks. We also showcase that Betty enables scaling MLO to models with hundreds of millions of parameters. We open-source the code at https://github.com/leopard-ai/betty.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

37 of 56 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  10. 2019Meta-Learning with Implicit Gradientscited by 523
  11. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  12. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
25 more
  1. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  2. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  3. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  4. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  5. 2018Approximation Methods for Bilevel Programmingcited by 231
  6. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
  7. 2021Efficient and Modular Implicit Differentiationcited by 199
  8. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  9. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  10. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  11. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  12. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 103
  13. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  14. 2019Generalized Inner Loop Meta-Learningcited by 92
  15. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  16. 2020learn2learn: A Library for Meta-Learning Researchcited by 69
  17. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 62
  18. 2020Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learningcited by 53
  19. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  20. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  21. 2020How Important is the Train-Validation Split in Meta-Learning?cited by 30
  22. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  23. 2021A Gradient Method for Multilevel Optimizationcited by 12
  24. 2020Meta-Semi: A Meta-learning Approach for Semi-supervised Learningcited by 11
  25. 2021Learning from Mistakes -- A Framework for Neural Architecture Searchcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 14; 2025: 2; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 30
  2. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  3. 2022TorchOpt: An Efficient Library for Differentiable Optimizationcited by 17
  4. 2023Making Scalable Meta Learning Practicalcited by 17
  5. 2023Nystrom Method for Accurate and Scalable Implicit Differentiationcited by 8
  6. 2024BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAMcited by 8
  7. 2024Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimescited by 8
  8. 2024BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Modelscited by 7
  9. 2023Provably Convergent Federated Trilevel Learningcited by 6
  10. 2023Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encodercited by 5
  11. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
  12. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
15 more
  1. 2024Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Textscited by 2
  2. 2025A stochastic gradient method for trilevel optimizationcited by 2
  3. 2024On the Universality of Self-Supervised Learningcited by 1
  4. 2024Rethinking Meta-Learning from a Learning Lenscited by 1
  5. 2024TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretrainingcited by 1
  6. 2024Consistency of Compositional Generalization across Multiple Levelscited by 1
  7. 2025DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Trainingcited by 1
  8. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  9. 2023AwesomeMeta+: A Mixed-Prototyping Meta-Learning System Supporting AI Application Design Anywherecited by 0
  10. 2024Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimizationcited by 0
  11. 2024Self-Supervised Video Representation Learning in a Heuristic Decoupled Perspectivecited by 0
  12. 2024Towards Differentiable Multilevel Optimization: A Gradient-Based Approachcited by 0
  13. 2026BLO-Inst: Bi-Level Optimization Based Alignment of YOLO and SAM for Robust Instance Segmentationcited by 0
  14. 2026Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimizationcited by 0
  15. 2026CHAL: Council of Hierarchical Agentic Languagecited by 0