mlsanityData to 14 Oct 2026

Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD

arXiv 2505.18535 ↗First posted 24 May 2025cs.LG math.PR stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the convergence properties and escape dynamics of Stochastic Gradient Descent (SGD) in one-dimensional landscapes, separately considering infinite- and finite-variance noise. Our main focus is to identify the time scales on which SGD reliably moves from an initial point to the local minimum in the same ''basin''. Under suitable conditions on the noise distribution, we prove that SGD converges to the basin's minimum unless the initial point lies too close to a local maximum. In that near-maximum scenario, we show that SGD can linger for a long time in its neighborhood. For initial points near a ''sharp'' maximum, we show that SGD does not remain stuck there, and we provide results to estimate the probability that it will reach each of the two neighboring minima. Overall, our findings present a nuanced view of SGD's transitions between local maxima and minima, influenced by both noise characteristics and the underlying function geometry.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

24 of 48 references.

References · 24

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  3. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  4. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  5. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  6. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  7. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  8. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 44
  9. 2019On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networkscited by 43
  10. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  11. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  12. 2021The effective noise of Stochastic Gradient Descentcited by 25
12 more
  1. 2024What is the long-run distribution of stochastic gradient descent? A large deviations analysiscited by 17
  2. 2021Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noisecited by 14
  3. 2019Accelerating Minibatch Stochastic Gradient Descent using Typicality Samplingcited by 11
  4. 2024Revisiting Step-Size Assumptions in Stochastic Approximationcited by 10
  5. 2022Large deviations rates for stochastic gradient descent with strongly convex functionscited by 9
  6. 2019Non-Asymptotic Analysis of Fractional Langevin Monte Carlo for Non-Convex Optimizationcited by 8
  7. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  8. 2023Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descentcited by 8
  9. 2022On uniform-in-time diffusion approximation for stochastic gradient descentcited by 6
  10. 2019Quasi-potential as an implicit regularizer for the loss function in the stochastic gradient descentcited by 3
  11. 2025A weak convergence approach to large deviations for stochastic approximationscited by 2
  12. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1

Built on by

Not cited yet.