mlsanityData to 7 Oct 2026

Theory of Deep Learning IIb: Optimization Properties of SGD

arXiv 1801.02254 ↗First posted 7 Jan 2018cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper is theoretical and experimental evidence for the following conjecture about SGD: SGD concentrates in probability -- like the classical Langevin equation -- on large volume, "flat" minima, selecting flat minimizers which are with very high probability also global minimizers

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 3; 2023: 1; 2024: 1; 2025: 1.

12 papers.

  1. 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 86
  2. 2021Strength of Minibatch Noise in SGDcited by 39
  3. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  4. 2022On the generalization of learning algorithms that do not convergecited by 9
  5. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  6. 2021What training reveals about neural network complexitycited by 5
  7. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  8. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  9. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  10. 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Netscited by 3
  11. 2021On the Distributional Properties of Adaptive Gradientscited by 0
  12. 2025Generalization of the Gibbs algorithm with high probability at low temperaturescited by 0