mlsanityData to 14 Oct 2026

Three Factors Influencing Minima in SGD

arXiv 1711.04623 ↗First posted 13 Nov 2017cs.LG cs.AI cs.CVcited by 273

Not on the map. Why?

Abstract (arXiv)

We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rate, batch size and the properties of the final minima, such as width or generalization, remains an open question. In order to tackle this problem we investigate the previously proposed approximation of SGD by a stochastic differential equation (SDE). We theoretically argue that three factors - learning rate, batch size and gradient covariance - influence the minima found by SGD. In particular we find that the ratio of learning rate to batch size is a key determinant of SGD dynamics and of the width of the final minima, and that higher values of the ratio lead to wider minima and often better generalization. We confirm these findings experimentally. Further, we include experiments which show that learning rate schedules can be replaced with batch size schedules and that the ratio of learning rate to batch size is an important factor influencing the memorization process.

Key facts. Cited by 273 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 47; 2022: 48; 2023: 61; 2024: 43; 2025: 36; 2026: 38.

273 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  3. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  4. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  5. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  6. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  7. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  8. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  9. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  10. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  11. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  12. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
38 more
  1. 2022Understanding the unstable convergence of gradient descentcited by 86
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  3. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  4. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
  5. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  6. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 73
  7. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  8. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  9. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  10. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  11. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  12. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  13. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  14. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  15. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40
  16. 2022How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularizationcited by 40
  17. 2021Strength of Minibatch Noise in SGDcited by 39
  18. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  19. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  20. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  21. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  22. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  23. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  24. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
  25. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 35
  26. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 33
  27. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  28. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  29. 2022The Effect of Task Ordering in Continual Learningcited by 31
  30. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  31. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  32. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  33. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  34. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  35. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 28
  36. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  37. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  38. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 27

50 most cited of 273 All 273 (JSON)