# A Dynamical Central Limit Theorem for Shallow Neural Networks

arXiv: https://arxiv.org/abs/2008.09623 · HTML: https://mlsanity.com/paper/2008.09623/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [Convex Analysis of the Mean Field Langevin Dynamics](https://mlsanity.com/paper/2201.10469/) (2022)
- [Feature-Learning Networks Are Consistent Across Widths At Realistic Scales](https://mlsanity.com/paper/2305.18411/) (2023)
- [Embedding Principle of Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2105.14573/) (2021)
- [Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction](https://mlsanity.com/paper/2306.07221/) (2023)
- [Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2107.04479/) (2021)
- [Gaussian random field approximation via Stein's method with applications to wide random neural networks](https://mlsanity.com/paper/2306.16308/) (2023)
- [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks](https://mlsanity.com/paper/2107.05134/) (2021)
- [A spectral-based analysis of the separation between two-layer neural networks and linear methods](https://mlsanity.com/paper/2108.04964/) (2021)
- [Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training](https://mlsanity.com/paper/2110.15954/) (2021)
- [Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence](https://mlsanity.com/paper/2210.06819/) (2022)
- [Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training](https://mlsanity.com/paper/2304.03385/) (2023)
- [BGDB: Bernoulli-Gaussian Decision Block with Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2409.13116/) (2024)
- [Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks](https://mlsanity.com/paper/2605.22010/) (2026)
- [Continuous-in-time Limit for Bayesian Bandits](https://mlsanity.com/paper/2210.07513/) (2022)
- [Symmetries in Overparametrized Neural Networks: A Mean-Field View](https://mlsanity.com/paper/2405.19995/) (2024)
- [TULiP: Test-time Uncertainty Estimation via Linearization and Weight Perturbation](https://mlsanity.com/paper/2505.16923/) (2025)
- [The Underlying Correlated Dynamics in Neural Training](https://mlsanity.com/paper/2212.09040/) (2022)
- [Central Limit Theorem for Bayesian Neural Network trained with Variational Inference](https://mlsanity.com/paper/2406.09048/) (2024)
- [The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearity](https://mlsanity.com/paper/2503.10587/) (2025)
- [AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery](https://mlsanity.com/paper/2504.01875/) (2025)
- [Fluctuations of Nonlinear Observables in Mean Field Neural Network Training](https://mlsanity.com/paper/2610.09768/) (2026)

## Abstract (arXiv)

Recent theoretical works have characterized the dynamics of wide shallow neural networks trained via gradient descent in an asymptotic mean-field limit when the width tends towards infinity. At initialization, the random sampling of the parameters leads to deviations from the mean-field limit dictated by the classical Central Limit Theorem (CLT). However, since gradient descent induces correlations among the parameters, it is of interest to analyze how these fluctuations evolve. Here, we use a dynamical CLT to prove that the asymptotic fluctuations around the mean limit remain bounded in mean square throughout training. The upper bound is given by a Monte-Carlo resampling error, with a variance that that depends on the 2-norm of the underlying measure, which also controls the generalization error. This motivates the use of this 2-norm as a regularization term during training. Furthermore, if the mean-field dynamics converges to a measure that interpolates the training data, we prove that the asymptotic deviation eventually vanishes in the CLT scaling. We also complement these results with numerical experiments.