Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
Not on the map. Why?
Abstract (arXiv)
Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep learning, a wide range of optimizers with different approaches have been developed. This study aims to provide a review of various optimizers that have been proposed and received attention in the literature. From Stochastic gradient descent to the most recent ones such as Momentum, AdamW, Sophia, and Muon in chronological order, optimizers are examined individually, and their distinctive features are highlighted in the study. The update rule of each optimizer is presented in detail, with an explanation of the associated concepts and variables. The techniques applied by these optimizers, their contributions to the optimization process, and their default hyperparameter settings are also discussed. In addition, insights are offered into the open challenges encountered in the optimization of deep learning models. Thus, a comprehensive resource is provided both for understanding the current state of optimizers and for identifying potential areas of future development.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 39 of its 71 references that are in the map. First posted on arXiv 22 Sep 2025.
Builds on
39 of 71 references.
References · 39
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Convex optimizationcited by 2,515
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2016An overview of gradient descent optimization algorithmscited by 1,033
- 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
- 2023Large Language Models as Optimizerscited by 864
- 2019On the Convergence of Adam and Beyondcited by 841
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
- 2017Large Batch Training of Convolutional Networkscited by 535
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
27 more
- 2023Symbolic Discovery of Optimization Algorithmscited by 434
- 2025Muon is Scalable for LLM Trainingcited by 428
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
- 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
- 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
- 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
- 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
- 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
- 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
- 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
- 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
- 2019A Survey of Optimization Methods from a Machine Learning Perspectivecited by 91
- 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
- 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
- 2019Optimization for deep learning: theory and algorithmscited by 70
- 2025AdaMuon: Adaptive Muon Optimizercited by 67
- 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
- 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
- 2019On the adequacy of untuned warmup for adaptive optimizationcited by 35
- 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
- 2020Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedulecited by 25
- 2021Ranger21: a synergistic deep learning optimizercited by 22
- 2021Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networkscited by 3
- 2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.