# Priors in Bayesian Deep Learning: A Review

arXiv: https://arxiv.org/abs/2105.06868 · HTML: https://mlsanity.com/paper/2105.06868/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Builds on 170 of its 269 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 30 of 170)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)

## Built on by (most cited first, 30 of 101)

- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Dangers of Bayesian Model Averaging under Covariate Shift](https://mlsanity.com/paper/2106.11905/) (2021)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations](https://mlsanity.com/paper/2201.01702/) (2022)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models](https://mlsanity.com/paper/2403.19137/) (2024)
- [On Stein Variational Neural Network Ensembles](https://mlsanity.com/paper/2106.10760/) (2021)
- [The Landscape of Unfolding with Machine Learning](https://mlsanity.com/paper/2404.18807/) (2024)
- [Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2405.03425/) (2024)
- [Neural Variational Gradient Descent](https://mlsanity.com/paper/2107.10731/) (2021)
- [Random-Set Neural Networks (RS-NN)](https://mlsanity.com/paper/2307.05772/) (2023)
- [Bayesian Diffusion Models for 3D Shape Reconstruction](https://mlsanity.com/paper/2403.06973/) (2024)
- [Adaptive variational Bayes: Optimality, computation and applications](https://mlsanity.com/paper/2109.03204/) (2021)
- [Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference](https://mlsanity.com/paper/2407.17033/) (2024)
- [Improving Neural Additive Models with Bayesian Principles](https://mlsanity.com/paper/2305.16905/) (2023)
- [Pathologies in priors and inference for Bayesian transformers](https://mlsanity.com/paper/2110.04020/) (2021)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [Scalable Bayesian Learning with posteriors](https://mlsanity.com/paper/2406.00104/) (2024)
- [Season combinatorial intervention predictions with Salt & Peper](https://mlsanity.com/paper/2404.16907/) (2024)
- [Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization](https://mlsanity.com/paper/2304.08309/) (2023)
- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)
- [Gaussian Pre-Activations in Neural Networks: Myth or Reality?](https://mlsanity.com/paper/2205.12379/) (2022)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Position: Epistemic uncertainty estimation methods are fundamentally incomplete](https://mlsanity.com/paper/2505.23506/) (2025)

## Abstract (arXiv)

While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.