mlsanityData to 7 Oct 2026

SGD with a Constant Large Learning Rate Can Converge to Local Maxima

arXiv 2107.11774 ↗First posted 25 Jul 2021cs.LG math.OC stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regimes that the previous works often assume, SGD can exhibit many strange and potentially undesirable behaviors. Specifically, we construct landscapes and data distributions such that (1) SGD converges to local maxima, (2) SGD escapes saddle points arbitrarily slowly, (3) SGD prefers sharp minima over flat ones, and (4) AMSGrad converges to local maxima. We also realize results in a minimal neural network-like example. Our results highlight the importance of simultaneously analyzing the minibatch sampling, discrete-time updates rules, and realistic landscapes to understand the role of SGD in deep learning.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 25 Jul 2021.

Builds on

37 of 56 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2019On the Convergence of Adam and Beyondcited by 834
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  8. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  10. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  11. 2017How to Escape Saddle Points Efficientlycited by 284
  12. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
25 more
  1. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  2. 2017Three Factors Influencing Minima in SGDcited by 270
  3. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  5. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  6. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  7. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  8. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  9. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  10. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  11. 2018A Walk with SGDcited by 74
  12. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  13. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  14. 2018Escaping Saddles with Stochastic Gradientscited by 66
  15. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 66
  16. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  17. 2021Strength of Minibatch Noise in SGDcited by 39
  18. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  19. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  20. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  21. 2019Negative eigenvalues of the Hessian in deep neural networkscited by 20
  22. 2017A Generic Approach for Escaping Saddle pointscited by 13
  23. 2018Theory of Deep Learning IIb: Optimization Properties of SGDcited by 12
  24. 2020Dynamic of Stochastic Gradient Descent with State-Dependent Noisecited by 12
  25. 2019Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 1; 2025: 1; 2026: 1.

8 papers.

  1. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  2. 2022Exact Solutions of a Deep Linear Networkcited by 24
  3. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  4. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  5. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  6. 2022A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamicscited by 3
  7. 2025Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGDcited by 0
  8. 2026Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetrycited by 0