# Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach

arXiv: https://arxiv.org/abs/1805.00915 · HTML: https://mlsanity.com/paper/1805.00915/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 30 of 106)

- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [A Spectral Condition for Feature Learning](https://mlsanity.com/paper/2310.17813/) (2023)
- [Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks](https://mlsanity.com/paper/2205.09653/) (2022)
- [Don't be lazy: CompleteP enables compute-efficient deep transformers](https://mlsanity.com/paper/2505.01618/) (2025)
- [Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs](https://mlsanity.com/paper/2206.00939/) (2022)
- [How Two-Layer Neural Networks Learn, One (Giant) Step at a Time](https://mlsanity.com/paper/2305.18270/) (2023)
- [Do Input Gradients Highlight Discriminative Features?](https://mlsanity.com/paper/2102.12781/) (2021)
- [The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents](https://mlsanity.com/paper/2402.03220/) (2024)
- [Learning threshold neurons via the "edge of stability"](https://mlsanity.com/paper/2212.07469/) (2022)
- [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022)
- [From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks](https://mlsanity.com/paper/2409.14623/) (2024)
- [Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning](https://mlsanity.com/paper/2406.06158/) (2024)
- [Feature-Learning Networks Are Consistent Across Widths At Realistic Scales](https://mlsanity.com/paper/2305.18411/) (2023)
- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [Asymptotics of feature learning in two-layer networks after one gradient-step](https://mlsanity.com/paper/2402.04980/) (2024)
- [Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions](https://mlsanity.com/paper/2405.15459/) (2024)
- [The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization](https://mlsanity.com/paper/2106.04013/) (2021)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Uniform-in-time propagation of chaos for mean field Langevin dynamics](https://mlsanity.com/paper/2212.03050/) (2022)
- [Understanding Deflation Process in Over-parametrized Tensor Decomposition](https://mlsanity.com/paper/2106.06573/) (2021)
- [Meta-Principled Family of Hyperparameter Scaling Strategies](https://mlsanity.com/paper/2210.04909/) (2022)
- [Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer](https://mlsanity.com/paper/2502.02531/) (2025)
- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization](https://mlsanity.com/paper/2502.03435/) (2025)
- [The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram](https://mlsanity.com/paper/2509.10167/) (2025)
- [A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities](https://mlsanity.com/paper/2410.18938/) (2024)
- [Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD](https://mlsanity.com/paper/2305.18502/) (2023)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [Optimal Protocols for Continual Learning via Statistical Physics and Control Theory](https://mlsanity.com/paper/2409.18061/) (2024)
- [Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks](https://mlsanity.com/paper/2506.06489/) (2025)

## Abstract (arXiv)

Neural networks, a central tool in machine learning, have demonstrated remarkable, high fidelity performance on image recognition and classification tasks. These successes evince an ability to accurately represent high dimensional functions, but rigorous results about the approximation error of neural networks after training are few. Here we establish conditions for global convergence of the standard optimization algorithm used in machine learning applications, stochastic gradient descent (SGD), and quantify the scaling of its error with the size of the network. This is done by reinterpreting SGD as the evolution of a particle system with interactions governed by a potential related to the objective or "loss" function used to train the network. We show that, when the number n of units is large, the empirical distribution of the particles descends on a convex landscape towards the global minimum at a rate independent of n, with a resulting approximation error that universally scales as O(n-1). These properties are established in the form of a Law of Large Numbers and a Central Limit Theorem for the empirical distribution. Our analysis also quantifies the scale and nature of the noise introduced by SGD and provides guidelines for the step size and batch size to use when training a neural network. We illustrate our findings on examples in which we train neural networks to learn the energy function of the continuous 3-spin model on the sphere. The approximation error scales as our analysis predicts in as high a dimension as d=25.