mlsanityData to 7 Oct 2026

Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysis

arXiv 2106.02588 ↗First posted 4 Jun 2021cs.LG math.AP stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

The representation of functions by artificial neural networks depends on a large number of parameters in a non-linear fashion. Suitable parameters of these are found by minimizing a 'loss functional', typically by stochastic gradient descent (SGD) or an advanced SGD-based algorithm. In a continuous time model for SGD with noise that follows the 'machine learning scaling', we show that in a certain noise regime, the optimization algorithm prefers 'flat' minima of the objective function in a sense which is different from the flat minimum selection of continuous time SGD with homogeneous noise.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 28 of its 81 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

28 of 81 references.

References · 28

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  6. 2017Three Factors Influencing Minima in SGDcited by 270
  7. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  8. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  10. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  11. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  12. 2020Implicit Gradient Regularizationcited by 145
16 more
  1. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  2. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  3. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  4. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  5. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  6. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  7. 2021Strength of Minibatch Noise in SGDcited by 39
  8. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  9. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  10. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
  11. 2020Analysis of Stochastic Gradient Descent in Continuous Timecited by 24
  12. 2019Mean-Field Neural ODEs via Relaxed Optimal Controlcited by 16
  13. 2016Stochastic Gradient Descent in Continuous Timecited by 14
  14. 2017Stochastic Gradient Descent in Continuous Time: A Central Limit Theoremcited by 10
  15. 2017The Impact of Local Geometry and Batch Size on Stochastic Gradient Descent for Nonconvex Problemscited by 2
  16. 2020How Many Factors Influence Minima in SGD?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 8; 2024: 4; 2025: 1; 2026: 2.

25 papers.

Sort:

  1. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  2. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  3. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  4. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  5. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  6. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  7. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  8. 2021On minimal representations of shallow ReLU networkscited by 13
  9. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  10. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  11. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  12. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
13 more
  1. 2021A Continuous-time Stochastic Gradient Descent Method for Continuous Datacited by 7
  2. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 7
  3. 2023Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descentcited by 7
  4. 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
  5. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  6. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  7. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  8. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Modelscited by 3
  9. 2024Stochastic Modified Flows for Riemannian Stochastic Gradient Descentcited by 2
  10. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  11. 2022Identical Image Retrieval using Deep Learningcited by 0
  12. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  13. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0