mlsanityData to 14 Oct 2026

Escaping Saddles with Stochastic Gradients

arXiv 1803.05999 ↗First posted 15 Mar 2018cs.LG math.OC stat.MLcited by 66

Not on the map. Why?

Abstract (arXiv)

We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show that - contrary to the case of isotropic noise - this variance is proportional to the magnitude of the corresponding eigenvalues and not decreasing in the dimensionality. Based upon this observation we propose a new assumption under which we show that the injection of explicit, isotropic noise usually applied to make gradient descent escape saddle points can successfully be replaced by a simple SGD step. Additionally - and under the same condition - we derive the first convergence rate for plain SGD to a second-order stationary point in a number of iterations that is independent of the problem dimension.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 14; 2023: 14; 2024: 9; 2025: 5; 2026: 6.

66 papers.

Sort:

By area:

  1. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  2. 2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
  3. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  4. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  5. 2022A Theoretical Analysis of the Learning Dynamics under Class Imbalancecited by 20
  6. 2022Efficiently Escaping Saddle Points in Bilevel Optimizationcited by 19
  7. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  8. 2023How to escape sharp minima with random perturbationscited by 13
  9. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  10. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  11. 2026FlashSinkhorn: IO-Aware Entropic Optimal Transport on GPUcited by 11
  12. 2021Provable Super-Convergence with a Large Cyclical Learning Ratecited by 10
38 more
  1. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  2. 2022Decentralized Nonconvex Optimization with Guaranteed Privacy and Accuracycited by 10
  3. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 9
  4. 2021Escaping Saddle Points Faster with Stochastic Momentumcited by 8
  5. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  6. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 8
  7. 2021Escaping strict saddle points of the Moreau envelope in nonsmooth optimizationcited by 7
  8. 2022Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Modelscited by 6
  9. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  10. 2021On the Convergence of Differentially Private Federated Learning on Non-Lipschitz Objectives, and with Normalized Client Updatescited by 5
  11. 2021On the Second-order Convergence Properties of Random Search Methodscited by 5
  12. 2021Exponential escape efficiency of SGD from sharp minima in non-stationary regimecited by 5
  13. 2022Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimizationcited by 5
  14. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
  15. 2023Efficient displacement convex optimization with particle gradient descentcited by 5
  16. 2023On the Second-Order Convergence of Biased Policy Gradient Algorithmscited by 5
  17. 2024Concurrent Training and Layer Pruning of Deep Neural Networkscited by 5
  18. 2021Escaping Saddle Points for Nonsmooth Weakly Convex Functions via Perturbed Proximal Algorithmscited by 4
  19. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  20. 2023Escaping Saddle Points in Heterogeneous Federated Learning via Distributed SGD with Communication Compressioncited by 4
  21. 2021Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networkscited by 3
  22. 2021Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)cited by 3
  23. 2023Convergence Analysis of Stochastic Gradient Descent with MCMC Estimatorscited by 3
  24. 2023Enhance Diffusion to Improve Robust Generalizationcited by 3
  25. 2023Max-affine regression via first-order methodscited by 3
  26. 2024SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAMcited by 3
  27. 2025Second-Order Convergence in Private Stochastic Non-Convex Optimizationcited by 3
  28. 2026Stochastic Generative Plug-and-Play Priorscited by 3
  29. 2022Tackling benign nonconvexity with smoothing and stochastic gradientscited by 2
  30. 2022Losing momentum in continuous-time stochastic optimisationcited by 2
  31. 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
  32. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 2
  33. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  34. 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2
  35. 2021On the Differentially Private Nature of Perturbed Gradient Descentcited by 1
  36. 2021Noisy Truncated SGD: Optimization and Generalizationcited by 1
  37. 2021Faster Perturbed Stochastic Gradient Methods for Finding Local Minimacited by 1
  38. 2022A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorizationcited by 1

50 most cited of 66 All 66 (JSON)