mlsanityData to 7 Oct 2026

MADA: Meta-Adaptive Optimizers through hyper-gradient Descent

arXiv 2401.08893 ↗First posted 17 Jan 2024cs.LG math.OCcited by 6

Not on the map. Why?

Abstract (arXiv)

Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Adam uniformly across all tasks. In this work, we introduce Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training. The key idea in MADA is to parameterize the space of optimizers and dynamically search through it using hyper-gradient descent during training. We empirically compare MADA to other popular optimizers on vision and language tasks, and find that MADA consistently outperforms Adam and other popular optimizers, and is robust against sub-optimally tuned hyper-parameters. MADA achieves a greater validation performance improvement over Adam compared to other popular optimizers during GPT-2 training and fine-tuning. We also propose AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization. Finally, we provide a convergence analysis to show that parameterized interpolations of optimizers can improve their error bounds (up to constants), hinting at an advantage for meta-optimizers.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

25 of 37 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2016Learning to learn by gradient descent by gradient descentcited by 616
  12. 2017Large Batch Training of Convolutional Networkscited by 529
13 more
  1. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  2. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  3. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  4. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  5. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  6. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  7. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  8. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  9. 2017Learned Optimizers that Scale and Generalizecited by 109
  10. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  11. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  12. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  13. 2021A Generalizable Approach to Learning Optimizerscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  2. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  3. 2025Provable and Practical Online Learning Rate Adaptation with Hypergradient Descentcited by 5
  4. 2025Gradient Methods with Online Scaling Part I. Theoretical Foundationscited by 3
  5. 2025Online Learning-guided Learning Rate Adaptation via Gradient Alignmentcited by 1
  6. 2026Anon: Extrapolating Adaptivity Beyond SGD and Adamcited by 0