mlsanityData to 7 Oct 2026

Symbolic Discovery of Optimization Algorithms

arXiv 2302.06675 ↗First posted 13 Feb 2023cs.LG cs.AI cs.CLcited by 423

Not on the map. Why?

Abstract (arXiv)

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, Lion (textitEvoLved Sign Momentum). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% textitzero-shot and 91.1% textitfine-tuning accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.

Key facts. Cited by 423 papers in the map (data through 7 Oct 2026). Builds on 78 of its 111 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

78 of 111 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  12. 2020Denoising Diffusion Implicit Modelscited by 9,655
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  13. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  14. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  15. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  16. 2019Natural Adversarial Examplescited by 1,401
  17. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  18. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  19. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  20. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  21. 2021Scaling Vision Transformerscited by 993
  22. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  23. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  24. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  25. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  26. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  27. 2019On the Convergence of Adam and Beyondcited by 834
  28. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  29. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  30. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  31. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  32. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  33. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  34. 2016Learning to learn by gradient descent by gradient descentcited by 616
  35. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  36. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  37. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  38. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 81; 2024: 99; 2025: 117; 2026: 126.

423 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  4. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  5. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  6. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  9. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  10. 2024Old Optimizer, New Norm: An Anthologycited by 173
  11. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  12. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 118
38 more
  1. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  2. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  3. 2023Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Modelscited by 86
  4. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  5. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  6. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  7. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  8. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  9. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  10. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  11. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  12. 2025Dion: Distributed Orthonormalized Updatescited by 64
  13. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  14. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  15. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  16. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  17. 2024Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biologycited by 52
  18. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  19. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  20. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  21. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  22. 2025AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformerscited by 49
  23. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  24. 2026AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimizationcited by 46
  25. 2026EvoX: Meta-Evolution for Automated Discoverycited by 45
  26. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  27. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  28. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  29. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  30. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  31. 2025Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networkscited by 41
  32. 2023Learning to Decode the Surface Code with a Recurrent, Transformer-Based Neural Networkcited by 40
  33. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  34. 2025In Search of Adam's Secret Saucecited by 40
  35. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  36. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  37. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  38. 2023Scaling MLPs: A Tale of Inductive Biascited by 37

50 most cited of 423 All 423 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavior and Representation in Open-Weight Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection
  2. 2026K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
  3. 2026Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods
  4. 2026BiFE: Search-Efficient Discovery of CPU-Only Branching Policies via LLM-based Bi-Fidelity Evolution
  5. 2026MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources
  6. 2026Kernel Autoresearch for Open-Ended Model Discovery
  7. 2019Exact Combinatorial Optimization with Graph Convolutional Neural Networks
  8. 2021Search Spaces for Neural Model Training