mlsanityData to 7 Oct 2026

Practical tradeoffs between memory, compute, and performance in learned optimizers

arXiv 2203.11860 ↗First posted 22 Mar 2022cs.LG cs.NE math.OCcited by 32

Not on the map. Why?

Abstract (arXiv)

Optimization plays a costly and crucial role in developing machine learning systems. In learned optimizers, the few hyperparameters of commonly used hand-designed optimizers, e.g. Adam or SGD, are replaced with flexible parametric functions. The parameters of these functions are then optimized so that the resulting learned optimizer minimizes a target loss on a chosen class of models. Learned optimizers can both reduce the number of required training steps and improve the final test loss. However, they can be expensive to train, and once trained can be expensive to use due to computational and memory overhead for the optimizer itself. In this work, we identify and quantify the design features governing the memory, compute, and performance trade-offs for many learned and hand-designed optimizers. We further leverage our analysis to construct a learned optimizer that is both faster and more memory efficient than previous work. Our model and training code are open source.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 43 of its 83 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

43 of 83 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  9. 2016An overview of gradient descent optimization algorithmscited by 1,021
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  12. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
31 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  2. 2016Learning to learn by gradient descent by gradient descentcited by 616
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  5. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  6. 2016Learning to Optimizecited by 195
  7. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  8. 2017Learned Optimizers that Scale and Generalizecited by 109
  9. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  10. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  11. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  12. 2017Generalizing Hamiltonian Monte Carlo with Neural Networkscited by 52
  13. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  14. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  15. 2017Learning to Optimize Neural Netscited by 41
  16. 2018Aggregated Momentum: Stability Through Passive Dampingcited by 35
  17. 2020Training Stronger Baselines for Learning to Optimizecited by 31
  18. 2019Learning an Adaptive Learning Rate Schedulecited by 29
  19. 2020Using a thousand optimization tasks to learn hyperparameter search strategiescited by 29
  20. 2019Learning to Optimize in Swarmscited by 23
  21. 2021A Generalizable Approach to Learning Optimizerscited by 23
  22. 2020Improved Adversarial Training via Learned Optimizercited by 22
  23. 2020Reverse engineering learned optimizers reveals known and novel mechanismscited by 19
  24. 2017Reinforcement Learning for Learning Rate Controlcited by 16
  25. 2018Meta-Learning for Stochastic Gradient MCMCcited by 16
  26. 2020Scalable Second Order Optimization for Deep Learningcited by 16
  27. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
  28. 2021Training Learned Optimizers with Randomly Initialized Learned Optimizerscited by 13
  29. 2021Learn2Hop: Learned Optimization on Rough Landscapescited by 12
  30. 2019Using learned optimizers to make models robust to input noisecited by 10
  31. 2016Using Deep Q-Learning to Control Optimization Hyperparameterscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 6; 2024: 9; 2025: 7; 2026: 3.

32 papers.

Sort:

  1. 2022Discovered Policy Optimisationcited by 103
  2. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  3. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  4. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  5. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  6. 2022evosax: JAX-based Evolution Strategiescited by 39
  7. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  8. 2023Improving physics-informed neural networks with meta-learned optimizationcited by 19
  9. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  10. 2022Transformer-Based Learned Optimizationcited by 13
  11. 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
  12. 2023Learning to Optimize for Reinforcement Learningcited by 12
20 more
  1. 2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
  2. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  3. 2024QN-Mixer: A Quasi-Newton MLP-Mixer Model for Sparse-View CT Reconstructioncited by 6
  4. 2023Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategiescited by 5
  5. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  6. 2023A Nonstochastic Control Approach to Optimizationcited by 4
  7. 2024Learning to Explore for Stochastic Gradient MCMCcited by 4
  8. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  9. 2025Scalable Meta-Learning via Mixed-Mode Differentiationcited by 4
  10. 2025Learning a Zeroth-Order Optimizer for Fine-Tuning LLMscited by 3
  11. 2023Meta-learning Optimizers for Communication-Efficient Learningcited by 2
  12. 2024From Learning to Optimize to Learning Optimization Algorithmscited by 2
  13. 2024A Generalization Result for Convergence in Learning-to-Optimizecited by 1
  14. 2025PyLO: Towards Accessible Learned Optimizers in PyTorchcited by 1
  15. 2026Celo2: Towards Learned Optimization Free Lunchcited by 1
  16. 2024Dynamic Memory Based Adaptive Optimizationcited by 0
  17. 2025L-SR1: Learned Symmetric-Rank-One Preconditioningcited by 0
  18. 2025Exploring the Relationship between Brain Hemisphere States and Frequency Bands through Classical Machine Learning and Deep Learning Optimization Techniques with Neurofeedbackcited by 0
  19. 2026When Descent Is Too Stable: Event-Triggered Hamiltonian Learning to Optimizecited by 0
  20. 2026Efficient Long-Horizon Learning for Learned Optimizationcited by 0