Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection
Not on the map. Why?
Abstract (arXiv)
Infinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximate a trained neural network with a Gaussian model with bounds on the approximation error. In this work, we present an algorithmic framework to approximate a neural network of finite width and depth, and with not necessarily i.i.d. parameters, with a mixture of Gaussian processes with error bounds on the approximation error. In particular, we consider the Wasserstein distance to quantify the closeness between probabilistic models and, by relying on tools from optimal transport and Gaussian processes, we iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Crucially, for any NN and ε >0 our approach is able to return a mixture of Gaussian processes that is ε-close to the NN at a finite set of input points. Furthermore, we rely on the differentiability of the resulting error bound to show how our approach can be employed to tune the parameters of a NN to mimic the functional behavior of a given Gaussian process, e.g., for prior selection in the context of Bayesian inference. We empirically investigate the effectiveness of our results on both regression and classification problems with various neural network architectures. Our experiments highlight how our results can represent an important step towards understanding neural network predictions and formally quantifying their uncertainty.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 77 references that are in the map. First posted on arXiv 26 Jul 2024.
Builds on
43 of 77 references.
References · 43
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2013Intriguing properties of neural networkscited by 5,114
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
- 2016Deep Information Propagationcited by 194
31 more
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
- 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
- 2017Bayesian Compression for Deep Learningcited by 116
- 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
- 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
- 2017Noisy Natural Gradient as Variational Inferencecited by 94
- 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
- 2019Non-Gaussian processes and neural networks at finite widthscited by 78
- 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
- 2019A Wasserstein-type distance in the space of Gaussian Mixture Modelscited by 75
- 2020All You Need is a Good Functional Prior for Bayesian Deep Learningcited by 61
- 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
- 2020Probabilistic Safety for Bayesian Neural Networkscited by 41
- 2014Avoiding pathologies in very deep networkscited by 33
- 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
- 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24
- 2018Gradient Sampling Methods for Nonsmooth Optimizationcited by 23
- 2019Finite size corrections for neural network Gaussian processescited by 22
- 2020The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networkscited by 20
- 2023Quantitative CLTs in Deep Neural Networkscited by 19
- 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
- 2021Large-width functional asymptotics for deep Gaussian neural networkscited by 14
- 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 12
- 2022On the Robustness of Bayesian Neural Networks to Adversarial Attackscited by 12
- 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 10
- 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 10
- 2023BNN-DP: Robustness Certification of Bayesian Neural Networks via Dynamic Programmingcited by 8
- 2023A Quantitative Functional Central Limit Theorem for Shallow Neural Networkscited by 8
- 2018Wasserstein variational gradient descent: From semi-discrete optimal transport to ensemble variational inferencecited by 3
Built on by
Citing papers per year: 2025: 3.
3 papers.