# Theory of Deep Learning IIb: Optimization Properties of SGD

arXiv: https://arxiv.org/abs/1801.02254 · HTML: https://mlsanity.com/paper/1801.02254/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2018.

## Built on by (most cited first, 12 of 12)

- [Multi-Objective Loss Balancing for Physics-Informed Deep Learning](https://mlsanity.com/paper/2110.09813/) (2021)
- [Strength of Minibatch Noise in SGD](https://mlsanity.com/paper/2102.05375/) (2021)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [On the generalization of learning algorithms that do not converge](https://mlsanity.com/paper/2208.07951/) (2022)
- [SGD with a Constant Large Learning Rate Can Converge to Local Maxima](https://mlsanity.com/paper/2107.11774/) (2021)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [What training reveals about neural network complexity](https://mlsanity.com/paper/2106.04186/) (2021)
- [Global Convergence of SGD On Two Layer Neural Nets](https://mlsanity.com/paper/2210.11452/) (2022)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets](https://mlsanity.com/paper/2309.09258/) (2023)
- [On the Distributional Properties of Adaptive Gradients](https://mlsanity.com/paper/2105.07222/) (2021)
- [Generalization of the Gibbs algorithm with high probability at low temperatures](https://mlsanity.com/paper/2502.11071/) (2025)

## Abstract (arXiv)

In Theory IIb we characterize with a mix of theory and experiments the optimization of deep convolutional networks by Stochastic Gradient Descent. The main new result in this paper is theoretical and experimental evidence for the following conjecture about SGD: SGD concentrates in probability -- like the classical Langevin equation -- on large volume, "flat" minima, selecting flat minimizers which are with very high probability also global minimizers