# Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit

arXiv: https://arxiv.org/abs/2010.07355 · HTML: https://mlsanity.com/paper/2010.07355/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2206.12543/) (2022)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Fast Neural Kernel Embeddings for General Activations](https://mlsanity.com/paper/2209.04121/) (2022)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processes](https://mlsanity.com/paper/2109.09690/) (2021)
- [Kernel Regression with Infinite-Width Neural Networks on Millions of Examples](https://mlsanity.com/paper/2303.05420/) (2023)
- [Additive Multi-Index Gaussian process modeling, with application to multi-physics surrogate modeling of the quark-gluon plasma](https://mlsanity.com/paper/2306.07299/) (2023)
- [Scale Mixtures of Neural Network Gaussian Processes](https://mlsanity.com/paper/2107.01408/) (2021)
- [Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel](https://mlsanity.com/paper/2205.07384/) (2022)
- [Uncertainty Quantification From Scaling Laws in Deep Neural Networks](https://mlsanity.com/paper/2503.05938/) (2025)

## Abstract (arXiv)

Modern deep learning models have achieved great success in predictive accuracy for many data modalities. However, their application to many real-world tasks is restricted by poor uncertainty estimates, such as overconfidence on out-of-distribution (OOD) data and ungraceful failing under distributional shift. Previous benchmarks have found that ensembles of neural networks (NNs) are typically the best calibrated models on OOD data. Inspired by this, we leverage recent theoretical advances that characterize the function-space prior of an ensemble of infinitely-wide NNs as a Gaussian process, termed the neural network Gaussian process (NNGP). We use the NNGP with a softmax link function to build a probabilistic model for multi-class classification and marginalize over the latent Gaussian outputs to sample from the posterior. This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue. We also examine the calibration of previous approaches to classification with the NNGP, which treat classification problems as regression to the one-hot labels. In this case the Bayesian posterior is exact, and we compare several heuristics to generate a categorical distribution over classes. We find these methods are well calibrated under distributional shift. Finally, we consider an infinite-width final layer in conjunction with a pre-trained embedding. This replicates the important practical use case of transfer learning and allows scaling to significantly larger datasets. As well as achieving competitive predictive accuracy, this approach is better calibrated than its finite width analogue.