# A Primer on Bayesian Neural Networks: Review and Debates

arXiv: https://arxiv.org/abs/2309.16314 · HTML: https://mlsanity.com/paper/2309.16314/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 170 of its 266 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 30 of 170)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)

## Built on by (most cited first, 27 of 27)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Generative Uncertainty in Diffusion Models](https://mlsanity.com/paper/2502.20946/) (2025)
- [Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queries](https://mlsanity.com/paper/2405.13907/) (2024)
- [Posterior and variational inference for deep neural networks with heavy-tailed weights](https://mlsanity.com/paper/2406.03369/) (2024)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Bayesian RG Flow in Neural Network Field Theories](https://mlsanity.com/paper/2405.17538/) (2024)
- [Inferring the Langevin Equation with Uncertainty via Bayesian Neural Networks](https://mlsanity.com/paper/2402.01338/) (2024)
- [The Ensemble Kalman Inversion Race](https://mlsanity.com/paper/2511.15853/) (2025)
- [Neural Network Prediction of Strong Lensing Systems with Domain Adaptation and Uncertainty Quantification](https://mlsanity.com/paper/2411.03334/) (2024)
- [Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality](https://mlsanity.com/paper/2506.19144/) (2025)
- [Exploring Diffusion Models' Corruption Stage in Few-Shot Fine-tuning and Mitigating with Bayesian Neural Networks](https://mlsanity.com/paper/2405.19931/) (2024)
- [Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networks](https://mlsanity.com/paper/2503.11808/) (2025)
- [Last Layer Hamiltonian Monte Carlo](https://mlsanity.com/paper/2507.08905/) (2025)
- [Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantification](https://mlsanity.com/paper/2507.21670/) (2025)
- [Bayesian Topological Convolutional Neural Nets](https://mlsanity.com/paper/2510.11704/) (2025)
- [Just rotate it! Uncertainty estimation in closed-source models via multiple queries](https://mlsanity.com/paper/2405.13864/) (2024)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [Label Uncertainty for Ultrasound Segmentation](https://mlsanity.com/paper/2508.15635/) (2025)
- [Comparison of Deterministic and Probabilistic Machine Learning Algorithms for Precise Dimensional Control and Uncertainty Quantification in Additive Manufacturing](https://mlsanity.com/paper/2509.16233/) (2025)
- [Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition](https://mlsanity.com/paper/2510.05006/) (2025)
- [Minimaxity and Admissibility of Bayesian Neural Networks](https://mlsanity.com/paper/2604.04673/) (2026)
- [Uncertainty-aware classification and triage of structural heart disease using electrocardiography and echocardiography metrics](https://mlsanity.com/paper/2605.22968/) (2026)
- [Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Training](https://mlsanity.com/paper/2605.27747/) (2026)
- [An Auto-Scaling Approach for Serverless Environments Based on a Multi-Expert Consensus Mechanism](https://mlsanity.com/paper/2607.15511/) (2026)
- [Uncertainty quantification for trustworthy deep learning: Methods and measures](https://mlsanity.com/paper/2607.28248/) (2026)
- [Uncertainty propagation in auto-regressive random neural network models](https://mlsanity.com/paper/2608.20483/) (2026)
- [Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales](https://mlsanity.com/paper/2610.09886/) (2026)

## Abstract (arXiv)

Neural networks have achieved remarkable performance across various problem domains, but their widespread applicability is hindered by inherent limitations such as overconfidence in predictions, lack of interpretability, and vulnerability to adversarial attacks. To address these challenges, Bayesian neural networks (BNNs) have emerged as a compelling extension of conventional neural networks, integrating uncertainty estimation into their predictive capabilities. This comprehensive primer presents a systematic introduction to the fundamental concepts of neural networks and Bayesian inference, elucidating their synergistic integration for the development of BNNs. The target audience comprises statisticians with a potential background in Bayesian methods but lacking deep learning expertise, as well as machine learners proficient in deep neural networks but with limited exposure to Bayesian statistics. We provide an overview of commonly employed priors, examining their impact on model behavior and performance. Additionally, we delve into the practical considerations associated with training and inference in BNNs. Furthermore, we explore advanced topics within the realm of BNN research, acknowledging the existence of ongoing debates and controversies. By offering insights into cutting-edge developments, this primer not only equips researchers and practitioners with a solid foundation in BNNs, but also illuminates the potential applications of this dynamic field. As a valuable resource, it fosters an understanding of BNNs and their promising prospects, facilitating further advancements in the pursuit of knowledge and innovation.