mlsanityData to 7 Oct 2026

Power-law escape rate of SGD

arXiv 2105.09557 ↗First posted 20 May 2021cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 29

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of SGD noise to derive a stochastic differential equation (SDE) with simpler additive noise by performing a random time change. Using this formalism, we show that the log loss barrier Deltalog L=log[L(θs)/L(θ^*)] between a local minimum θ^* and a saddle θs determines the escape rate of SGD from the local minimum, contrary to the previous results borrowing from physics that the linear loss barrier Delta L=L(θs)-L(θ^*) decides the escape rate. Our escape-rate formula strongly depends on the typical magnitude h^* and the number n of the outlier eigenvalues of the Hessian. This result explains an empirical fact that SGD prefers flat minima with low effective dimensions, giving an insight into implicit biases of SGD.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 17 of its 46 references that are in the map. First posted on arXiv 20 May 2021.

Builds on

17 of 46 references.

References · 17

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  5. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  6. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  7. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  8. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  9. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  10. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  11. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  12. 2020The Heavy-Tail Phenomenon in SGDcited by 122
5 more
  1. 2018Discovering physical concepts with neural networkscited by 117
  2. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
  3. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  4. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  5. 2020Dynamic of Stochastic Gradient Descent with State-Dependent Noisecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 7; 2025: 5; 2026: 9.

29 papers.

Sort:

By area:

  1. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  2. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  3. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  4. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  5. 2022Exact Solutions of a Deep Linear Networkcited by 24
  6. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  7. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  8. 2024Do Parameters Reveal More than Loss for Membership Inference?cited by 9
  9. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  10. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  11. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 6
  12. 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
17 more
  1. 2024Emergence of heavy tails in homogenized stochastic gradient descentcited by 4
  2. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 4
  3. 2024On the Trade-off between Flatness and Optimization in Distributed Learningcited by 3
  4. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  5. 2025Reasoning Bias of Next Token Prediction Trainingcited by 3
  6. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Modelscited by 3
  7. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  8. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  9. 2025Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shiftscited by 1
  10. 2025On the Escaping Efficiency of Distributed Adversarial Training Algorithmscited by 0
  11. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  12. 2026Classical and Quantum Speedups for Non-Convex Optimization via Energy Conserving Descentcited by 0
  13. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0
  14. 2026How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Lawscited by 0
  15. 2026Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlocited by 0
  16. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  17. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0