mlsanityData to 7 Oct 2026

VeLO: Training Versatile Learned Optimizers by Scaling Up

arXiv 2211.09760 ↗First posted 17 Nov 2022cs.LG math.OC stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

While deep learning models have replaced hand-designed features across many domains, these models are still trained with hand-designed optimizers. In this work, we leverage the same scaling approach behind the success of deep learning to learn versatile optimizers. We train an optimizer for deep learning which is itself a small neural network that ingests gradients and outputs parameter updates. Meta-trained with approximately four thousand TPU-months of compute on a wide variety of optimization tasks, our optimizer not only exhibits compelling performance, but optimizes in interesting and unexpected ways. It requires no hyperparameter tuning, instead automatically adapting to the specifics of the problem being optimized. We open source our learned optimizer, meta-training code, the associated train and test data, and an extensive optimizer benchmark suite with baselines at velo-code.github.io.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 90 of its 153 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

90 of 153 references.

References · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2017Proximal Policy Optimization Algorithmscited by 14,441
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  13. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  14. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  15. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  16. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  17. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  18. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  19. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  20. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  21. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  22. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  23. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  24. 2016Learning to learn by gradient descent by gradient descentcited by 616
  25. 2017Self-Normalizing Neural Networkscited by 591
  26. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  27. 2017Large Batch Training of Convolutional Networkscited by 529
  28. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  29. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  30. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  31. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  32. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  33. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  34. 2018An Empirical Model of Large-Batch Trainingcited by 268
  35. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  36. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  37. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  38. 2022Multi-Game Decision Transformerscited by 212

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 19; 2025: 9; 2026: 9.

57 papers.

Sort:

By area:

  1. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  2. 2023Permutation Equivariant Neural Functionalscited by 69
  3. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  4. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  5. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  6. 2023Neural Functional Transformerscited by 50
  7. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  8. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  9. 2023Learning to Warm-Start Fixed-Point Optimization Algorithmscited by 32
  10. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  11. 2024Universal Neural Functionalscited by 25
  12. 2025Evolution Strategies at the Hyperscalecited by 22
38 more
  1. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  2. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  3. 2023Making Scalable Meta Learning Practicalcited by 17
  4. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  5. 2023Learning to Optimize for Reinforcement Learningcited by 12
  6. 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
  7. 2025ReSplat: Learning Recurrent Gaussian Splattingcited by 12
  8. 2024Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalizationcited by 10
  9. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  10. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 8
  11. 2023Graph Reinforcement Learning for Network Control via Bi-Level Optimizationcited by 7
  12. 2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
  13. 2024MADA: Meta-Adaptive Optimizers through hyper-gradient Descentcited by 6
  14. 2023MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learningcited by 5
  15. 2024Moco: A Learnable Meta Optimizer for Combinatorial Optimizationcited by 5
  16. 2024Data-Driven Performance Guarantees for Classical and Learned Optimizerscited by 5
  17. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  18. 2023A Nonstochastic Control Approach to Optimizationcited by 4
  19. 2024Learning to Explore for Stochastic Gradient MCMCcited by 4
  20. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  21. 2024AutoFT: Learning an Objective for Robust Fine-Tuningcited by 3
  22. 2024Discovering Minimal Reinforcement Learning Environmentscited by 3
  23. 2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
  24. 2023Federated Automatic Differentiationcited by 2
  25. 2023Meta-learning Optimizers for Communication-Efficient Learningcited by 2
  26. 2023Generalisable Agents for Neural Network Optimisationcited by 1
  27. 2024Learning to Learn without Forgetting using Attentioncited by 1
  28. 2025Make Optimization Once and for All with Fine-grained Guidancecited by 1
  29. 2025PyLO: Towards Accessible Learned Optimizers in PyTorchcited by 1
  30. 2026Celo2: Towards Learned Optimization Free Lunchcited by 1
  31. 2026Quasi-Equivariant Metanetworkscited by 1
  32. 2026Learn2Splat: Extending the Horizon of Learned 3DGS Optimizationcited by 1
  33. 2026AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipescited by 1
  34. 2026Dynamic Neural Graph Encoding of Inference Processes in Deep Weight Spacecited by 1
  35. 2023Mathematical Challenges in Deep Learningcited by 0
  36. 2023HUB: Guiding Learned Optimizers with Continuous Prompt Tuningcited by 0
  37. 2024Dynamic Memory Based Adaptive Optimizationcited by 0
  38. 2024Implicit Neural Image Field for Biological Microscopy Image Compressioncited by 0

50 most cited of 57 All 57 (JSON)