Data to 14 Oct 2026

A Generalizable Approach to Learning Optimizers

arXiv 2106.00958 ↗First posted 2 Jun 2021cs.LG cs.AI cs.NEcited by 23

Not on the map. Why?

Abstract (arXiv)

A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalization-first perspective, learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2x speedups on ImageNet, and a 2.5x speedup on a language modeling task using over 5 orders of magnitude more compute than the training tasks.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

43 of 55 references.

References · 43

2017Attention Is All You Needcited by 60,361
2015Deep Residual Learning for Image Recognitioncited by 52,096
2014Adam: A Method for Stochastic Optimizationcited by 37,240
2020Language Models are Few-Shot Learnerscited by 32,419
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2017Proximal Policy Optimization Algorithmscited by 15,215
2020Scaling Laws for Neural Language Modelscited by 6,586
2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
2016Pointer Sentinel Mixture Modelscited by 3,096
2017Decoupled Weight Decay Regularizationcited by 1,831

31 more

2017Neural Collaborative Filteringcited by 1,111
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
2015Cyclical Learning Rates for Training Neural Networkscited by 629
2016Learning to learn by gradient descent by gradient descentcited by 620
2017Large Batch Training of Convolutional Networkscited by 535
2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
2018Deep Learning for Classical Japanese Literaturecited by 422
2017Population Based Training of Neural Networkscited by 367
2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
2018An Empirical Model of Large-Batch Trainingcited by 281
2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
2020On the Iteration Complexity of Hypergradient Computationcited by 155
2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
2019MLPerf Training Benchmarkcited by 115
2017Learned Optimizers that Scale and Generalizecited by 110
2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 58
2020The Two Regimes of Deep Network Trainingcited by 30
2019Learning an Adaptive Learning Rate Schedulecited by 29
2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 26
2020Scalable Second Order Optimization for Deep Learningcited by 21
2017Reinforcement Learning for Learning Rate Controlcited by 16
2015Stop Wasting My Gradients: Practical SVRGcited by 14
2019First-Order Preconditioning via Hypergradient Descentcited by 8
2015GradNets: Dynamic Interpolation Between Neural Architecturescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 4; 2024: 5; 2025: 4; 2026: 1.

23 papers.

Sort:

By area:

2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 86
2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
2022Automated Dynamic Algorithm Configurationcited by 39
2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
2022Meta-Gradients in Non-Stationary Environmentscited by 11
2025Vintix: Action Model via In-Context Reinforcement Learningcited by 10
2023Learning to Generalize Provably in Learning to Optimizecited by 8
2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
2024MADA: Meta-Adaptive Optimizers through hyper-gradient Descentcited by 6

11 more

2024Data-Driven Performance Guarantees for Classical and Learned Optimizerscited by 5
2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
2021ModelPred: A Framework for Predicting Trained Model from Training Datacited by 2
2022Reinforcement Teachingcited by 1
2023Generalisable Agents for Neural Network Optimisationcited by 1
2026Celo2: Towards Learned Optimization Free Lunchcited by 1
2023Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillationcited by 0
2023Sentiment analysis in Tourism: Fine-tuning BERT or sentence embeddings concatenation?cited by 0
2025Towards Robust Learning to Optimize with Theoretical Guaranteescited by 0