mlsanityData to 7 Oct 2026

Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent

arXiv 2303.13093 ↗First posted 23 Mar 2023cs.LG math.OC physics.data-ancited by 8

Not on the map. Why?

Abstract (arXiv)

Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks can be divided into two types, among which the Type-II saddles are especially difficult to escape from because the gradient noise vanishes at the saddle. The dynamics of SGD around these saddles are thus to leading order described by a random matrix product process, and it is thus natural to study the dynamics of SGD around these saddles using the notion of probabilistic stability and the related Lyapunov exponent. Theoretically, we link the study of SGD dynamics to well-known concepts in ergodic theory, which we leverage to show that saddle points can be either attractive or repulsive for SGD, and its dynamics can be classified into four different phases, depending on the signal-to-noise ratio in the gradient close to the saddle.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

20 of 36 references.

References · 20

  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  4. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  5. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  7. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  8. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  9. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  10. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  11. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  12. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
8 more
  1. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  2. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  3. 2022spred: Solving L1 Penalty with SGDcited by 16
  4. 2021Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspectivecited by 15
  5. 2021Smooth Bilevel Programming for Sparse Regularizationcited by 14
  6. 2019Minimal model of permutation symmetry in unsupervised learningcited by 13
  7. 2019Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex Optimizationcited by 11
  8. 2022Smooth over-parameterized solvers for non-smooth structured optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 2.

8 papers.

  1. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  2. 2023Exact Mean Square Linear Stability Analysis for SGDcited by 10
  3. 2024Dynamical stability and chaos in artificial neural network trajectories along trainingcited by 6
  4. 2024Convergence of continuous-time stochastic gradient descent with applications to deep neural networkscited by 4
  5. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2
  6. 2025Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGDcited by 0
  7. 2026Zeroth-Order Optimization at the Edge of Stabilitycited by 0
  8. 2026Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGDcited by 0