# Quantitative Propagation of Chaos for SGD in Wide Neural Networks

arXiv: https://arxiv.org/abs/2007.06352 · HTML: https://mlsanity.com/paper/2007.06352/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 20 of 20)

- [Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks](https://mlsanity.com/paper/2106.03795/) (2021)
- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction](https://mlsanity.com/paper/2306.07221/) (2023)
- [Interacting Particle Langevin Algorithm for Maximum Marginal Likelihood Estimation](https://mlsanity.com/paper/2303.13429/) (2023)
- [Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks](https://mlsanity.com/paper/2111.02278/) (2021)
- [Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems](https://mlsanity.com/paper/2312.01127/) (2023)
- [Wide stochastic networks: Gaussian limit and PAC-Bayesian training](https://mlsanity.com/paper/2106.09798/) (2021)
- [Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study](https://mlsanity.com/paper/2207.03859/) (2022)
- [Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD](https://mlsanity.com/paper/2306.08125/) (2023)
- [A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks](https://mlsanity.com/paper/2210.16286/) (2022)
- [Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks](https://mlsanity.com/paper/2605.22010/) (2026)
- [Symmetries in Overparametrized Neural Networks: A Mean-Field View](https://mlsanity.com/paper/2405.19995/) (2024)
- [Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference](https://mlsanity.com/paper/2307.04779/) (2023)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [Painless step size adaptation for SGD](https://mlsanity.com/paper/2102.00853/) (2021)
- [Neurons learn slower than they think](https://mlsanity.com/paper/2104.02578/) (2021)
- [Central Limit Theorem for Bayesian Neural Network trained with Variational Inference](https://mlsanity.com/paper/2406.09048/) (2024)
- [The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearity](https://mlsanity.com/paper/2503.10587/) (2025)
- [Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments](https://mlsanity.com/paper/2508.16001/) (2025)

## Abstract (arXiv)

In this paper, we investigate the limiting behavior of a continuous-time counterpart of the Stochastic Gradient Descent (SGD) algorithm applied to two-layer overparameterized neural networks, as the number or neurons (ie, the size of the hidden layer) N → +∞. Following a probabilistic approach, we show 'propagation of chaos' for the particle system defined by this continuous-time dynamics under different scenarios, indicating that the statistical interaction between the particles asymptotically vanishes. In particular, we establish quantitative convergence with respect to N of any particle to a solution of a mean-field McKean-Vlasov equation in the metric space endowed with the Wasserstein distance. In comparison to previous works on the subject, we consider settings in which the sequence of stepsizes in SGD can potentially depend on the number of neurons and the iterations. We then identify two regimes under which different mean-field limits are obtained, one of them corresponding to an implicitly regularized version of the minimization problem at hand. We perform various experiments on real datasets to validate our theoretical results, assessing the existence of these two regimes on classification problems and illustrating our convergence results.