mlsanityData to 7 Oct 2026

A survey of deep learning optimizers -- first and second order methods

arXiv 2211.15596 ↗First posted 28 Nov 2022cs.LG cs.CV math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep Learning optimization involves minimizing a high-dimensional loss function in the weight space which is often perceived as difficult due to its inherent difficulties such as saddle points, local minima, ill-conditioning of the Hessian and limited compute resources. In this paper, we provide a comprehensive review of 14 standard optimization methods successfully used in deep learning research and a theoretical assessment of the difficulties in numerical optimization from the optimization literature.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 22 of its 65 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

22 of 65 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2021Convex optimizationcited by 2,479
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  8. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  9. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  10. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  11. 2014The Loss Surfaces of Multilayer Networkscited by 278
  12. 2017Three Factors Influencing Minima in SGDcited by 270
10 more
  1. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  2. 2014Qualitatively characterizing neural network optimization problemscited by 186
  3. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  4. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  5. 2021Gradients are Not All You Needcited by 82
  6. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  7. 2021Learning in High Dimension Always Amounts to Extrapolationcited by 62
  8. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32
  9. 2014On the saddle point problem for non-convex optimizationcited by 19
  10. 2014Explorations on high dimensional landscapescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  2. 2025Accelerated Training of Federated Learning via Second-Order Methodscited by 0
  3. 2026Efficient Unlearning through Maximizing Relearning Convergence Delaycited by 0