On Bridging the Gap between Mean Field and Finite Width in Deep Random Neural Networks with Batch Normalization
Not on the map. Why?
Abstract (arXiv)
Mean field theory is widely used in the theoretical studies of neural networks. In this paper, we analyze the role of depth in the concentration of mean-field predictions, specifically for deep multilayer perceptron (MLP) with batch normalization (BN) at initialization. By scaling the network width to infinity, it is postulated that the mean-field predictions suffer from layer-wise errors that amplify with depth. We demonstrate that BN stabilizes the distribution of representations that avoids the error propagation of mean-field predictions. This stabilization, which is characterized by a geometric mixing property, allows us to establish concentration bounds for mean field predictions in infinitely-deep neural networks with a finite width.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 36 references that are in the map. First posted on arXiv 25 May 2022.
Builds on
17 of 36 references.
References · 17
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Layer Normalizationcited by 5,579
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2017Self-Normalizing Neural Networkscited by 591
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
- 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
5 more
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2018The Emergence of Spectral Universality in Deep Networkscited by 74
- 2019A Mean Field Theory of Batch Normalizationcited by 58
- 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
- 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.