Stochastic Neural Networks with Infinite Width are Deterministic
Not on the map. Why?
Abstract (arXiv)
This work theoretically studies stochastic neural networks, a main type of neural network in use. We prove that as the width of an optimized stochastic neural network tends to infinity, its predictive variance on the training set decreases to zero. Our theory justifies the common intuition that adding stochasticity to the model can help regularize the model by introducing an averaging effect. Two common examples that our theory can be relevant to are neural networks with dropout and Bayesian latent variable models in a special limit. Our result thus helps better understand how stochasticity affects the learning of neural networks and potentially design better architectures for practical problems.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 30 Jan 2022.
Builds on
35 of 51 references.
References · 35
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
- 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
- 2015Deep Kernel Learningcited by 440
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2018Understanding disentangling in β-VAEcited by 332
23 more
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
- 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
- 2016Deep Learning without Poor Local Minimacited by 233
- 2019Diagnosing and Enhancing VAE Modelscited by 192
- 2013Expectation Propagation for approximate Bayesian inferencecited by 172
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2019Practical Deep Learning with Bayesian Principlescited by 158
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
- 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
- 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
- 2016Identity Matters in Deep Learningcited by 107
- 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 97
- 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
- 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
- 2021Strength of Minibatch Noise in SGDcited by 39
- 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
- 2020Dropout: Explicit Forms and Capacity Controlcited by 27
- 2016Learning Scalable Deep Kernels with Recurrent Structurecited by 22
- 2020Wide Neural Networks with Bottlenecks are Deep Gaussian Processescited by 18
- 2017Dropout as a Low-Rank Regularizer for Matrix Factorizationcited by 14
- 2019On Dropout and Nuclear Norm Regularizationcited by 12
- 2021Variational autoencoders in the presence of low-dimensional data: landscape and implicit biascited by 11
Built on by
Citing papers per year: 2022: 2; 2023: 0; 2024: 1.
3 papers.