mlsanityData to 14 Oct 2026

Priors in Bayesian Deep Learning: A Review

arXiv 2105.06868 ↗First posted 14 May 2021stat.ML cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 170 of its 269 references that are in the map. First posted on arXiv 14 May 2021.

Builds on

170 of 269 references.

References · 170

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2015Variational Inference with Normalizing Flowscited by 2,170
  6. 2016Density estimation using Real NVPcited by 2,042
  7. 2016Variational Inference: A Review for Statisticianscited by 1,472
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  10. 2016HyperNetworkscited by 1,027
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  12. 2018On Lazy Training in Differentiable Programmingcited by 675
38 more
  1. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  2. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2015Deep Kernel Learningcited by 445
  7. 2018Conditional Neural Processescited by 426
  8. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  9. 2013Gaussian Processes for Big Datacited by 383
  10. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  11. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
  12. 2016Ladder Variational Autoencoderscited by 318
  13. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  14. 2018Neural Processescited by 311
  15. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  16. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  17. 2019Attentive Neural Processescited by 283
  18. 2018Probabilistic Model-Agnostic Meta-Learningcited by 256
  19. 2017VAE with a VampPriorcited by 248
  20. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  21. 2016Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoderscited by 220
  22. 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 220
  23. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  24. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  25. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  26. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  27. 2018Hyperspherical Variational Auto-Encoderscited by 189
  28. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  29. 2016Variational Lossy Autoencodercited by 176
  30. 2019GP-VAE: Deep Probabilistic Time Series Imputationcited by 172
  31. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  32. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  33. 2019Practical Deep Learning with Bayesian Principlescited by 158
  34. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  35. 2017Deep Exploration via Randomized Value Functionscited by 156
  36. 2017Stein Variational Gradient Descent as Gradient Flowcited by 152
  37. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  38. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 150

50 most cited of 170 All 170 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 11; 2023: 22; 2024: 27; 2025: 17; 2026: 10.

101 papers.

Sort:

By area:

  1. 2022Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisonscited by 170
  2. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  3. 2024Variational Bayesian Last Layerscited by 69
  4. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  5. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  6. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  7. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  8. 2024On the Challenges and Opportunities in Generative AIcited by 31
  9. 2022Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentationscited by 28
  10. 2023Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorialcited by 27
  11. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  12. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
38 more
  1. 2021On Stein Variational Neural Network Ensemblescited by 21
  2. 2024The Landscape of Unfolding with Machine Learningcited by 20
  3. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  4. 2021Neural Variational Gradient Descentcited by 19
  5. 2023Random-Set Neural Networks (RS-NN)cited by 19
  6. 2024Bayesian Diffusion Models for 3D Shape Reconstructioncited by 19
  7. 2021Adaptive variational Bayes: Optimality, computation and applicationscited by 17
  8. 2024Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inferencecited by 16
  9. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  10. 2021Pathologies in priors and inference for Bayesian transformerscited by 13
  11. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  12. 2024Scalable Bayesian Learning with posteriorscited by 13
  13. 2024Season combinatorial intervention predictions with Salt & Pepercited by 11
  14. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  15. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  16. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  17. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  18. 2025Position: Epistemic uncertainty estimation methods are fundamentally incompletecited by 9
  19. 2021Deep Classifiers with Label Noise Modeling and Distance Awarenesscited by 8
  20. 2023Bayesian neural networks via MCMC: a Python-based tutorialcited by 8
  21. 2024Bayesian Neural Networks with Domain Knowledge Priorscited by 8
  22. 2025Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'cited by 8
  23. 2021Bayesian Neural Networks: Essentialscited by 7
  24. 2023Solution of physics-based inverse problems using conditional generative adversarial networks with full gradient penaltycited by 7
  25. 2023Bayesian Gaussian Process ODEs via Double Normalizing Flowscited by 7
  26. 2021On out-of-distribution detection with Bayesian neural networkscited by 6
  27. 2022Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimizationcited by 6
  28. 2024Semi-Supervised Learning guided by the Generalized Bayes Rule under Soft Revisioncited by 6
  29. 2022Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical studycited by 5
  30. 2024Enhancing Transfer Learning with Flexible Nonparametric Posterior Samplingcited by 5
  31. 2024Low-Budget Simulation-Based Inference with Bayesian Neural Networkscited by 5
  32. 2025laplax -- Laplace Approximations with JAXcited by 5
  33. 2021Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Modelscited by 4
  34. 2023Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networkscited by 4
  35. 2024A Non-negative VAE:the Generalized Gamma Belief Networkcited by 4
  36. 2024Minimax optimality of deep neural networks on dependent data via PAC-Bayes boundscited by 4
  37. 2021Greedy Bayesian Posterior Approximation with Deep Ensemblescited by 3
  38. 2023Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variancecited by 3

50 most cited of 101 All 101 (JSON)