mlsanityData to 14 Oct 2026

Optimization for deep learning: theory and algorithms

arXiv 1912.08957 ↗First posted 19 Dec 2019cs.LG math.OC stat.MLcited by 70

Not on the map. Why?

Abstract (arXiv)

When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 16; 2023: 11; 2024: 11; 2025: 5; 2026: 4.

70 papers.

Sort:

By area:

  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  3. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
  4. 2021Model Complexity of Deep Learning: A Surveycited by 65
  5. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 50
  6. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  7. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 28
  8. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  9. 2022Convergence of gradient descent for deep neural networkscited by 24
  10. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  11. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  12. 2021Facial Emotion Recognition: State of the Art Performance on FER2013cited by 20
38 more
  1. 2021The dilemma of quantum neural networkscited by 20
  2. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
  3. 2021A Probabilistic Approach to Neural Network Pruningcited by 15
  4. 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 15
  5. 2024Convergence rates for the Adam optimizercited by 15
  6. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  7. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  8. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  9. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  10. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  11. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  12. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  13. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  14. 2025Sharp higher order convergence rates for the Adam optimizercited by 8
  15. 2022Stability and Risk Bounds of Iterative Hard Thresholdingcited by 7
  16. 2022NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learningcited by 7
  17. 2023Cut your Losses with Squentropycited by 6
  18. 2021NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learningcited by 5
  19. 2022Bounded nonlinear forecasts of partially observed geophysical systems with physics-constrained deep learningcited by 5
  20. 2021First-Order Methods for Convex Optimizationcited by 4
  21. 2021Accelerating variational quantum algorithms with multiple quantum processorscited by 4
  22. 2021Momentum Centering and Asynchronous Update for Adaptive Gradient Methodscited by 4
  23. 2022Defense against Backdoor Attacks via Identifying and Purifying Bad Neuronscited by 4
  24. 2024Fixed width treelike neural networks capacity analysis -- generic activationscited by 4
  25. 2021Deep learning for solution and inversion of structural mechanics and vibrationscited by 3
  26. 2022Adaptive neighborhood Metric learningcited by 3
  27. 2023Capacity of the treelike sign perceptrons neural networks with one hidden layer -- RDT based upper boundscited by 3
  28. 2023Lifted RDT based capacity analysis of the 1-hidden layer treelike sign perceptrons neural networkscited by 3
  29. 2024Foundations and Frontiers of Graph Learning Theorycited by 3
  30. 2021DeepMoM: Robust Deep Learning With Median-of-Meanscited by 2
  31. 2021Advancing biological super-resolution microscopy through deep learning: a brief reviewcited by 2
  32. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networkscited by 2
  33. 2022Stochastic Modeling of Inhomogeneities in the Aortic Wall and Uncertainty Quantification using a Bayesian Encoder-Decoder Surrogatecited by 2
  34. 2022Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theorycited by 2
  35. 2023Can Learning Be Explained By Local Optimality In Robust Low-rank Matrix Recovery?cited by 2
  36. 2023Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimizationcited by 2
  37. 2025Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rulescited by 2
  38. 2026Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gapscited by 2

50 most cited of 70 All 70 (JSON)