# Bayesian Learning for Neural Networks: an algorithmic survey

arXiv: https://arxiv.org/abs/2211.11865 · HTML: https://mlsanity.com/paper/2211.11865/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 142 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Mixed-curvature Variational Autoencoders](https://mlsanity.com/paper/1911.08411/) (2019)
- [A Tutorial on Learning With Bayesian Networks](https://mlsanity.com/paper/2002.00269/) (2020)
- [Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis](https://mlsanity.com/paper/1712.00975/) (2017)
- [Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models](https://mlsanity.com/paper/1807.04489/) (2018)
- [A Brief Introduction to Manifold Optimization](https://mlsanity.com/paper/1906.05450/) (2019)
- [Handling the Positive-Definite Constraint in the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10060/) (2020)
- [Variational Bayesian dropout: pitfalls and fixes](https://mlsanity.com/paper/1807.01969/) (2018)
- [Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift](https://mlsanity.com/paper/2006.14988/) (2020)
- [VIBNN: Hardware Acceleration of Bayesian Neural Networks](https://mlsanity.com/paper/1802.00822/) (2018)
- [Still no free lunches: the price to pay for tighter PAC-Bayes bounds](https://mlsanity.com/paper/1910.04460/) (2019)
- [Deep Gaussian Processes: A Survey](https://mlsanity.com/paper/2106.12135/) (2021)
- [A practical tutorial on Variational Bayes](https://mlsanity.com/paper/2103.01327/) (2021)
- [Variational Bayes on Manifolds](https://mlsanity.com/paper/1908.03097/) (2019)
- [Continuous Dropout](https://mlsanity.com/paper/1911.12675/) (2019)

## Built on by (most cited first, 23 of 23)

- [Bayesian RG Flow in Neural Network Field Theories](https://mlsanity.com/paper/2405.17538/) (2024)
- [Efficient Training of Probabilistic Neural Networks for Survival Analysis](https://mlsanity.com/paper/2404.06421/) (2024)
- [Machine Learning Methods for Small Data and Upstream Bioprocessing Applications: A Comprehensive Review](https://mlsanity.com/paper/2506.12322/) (2025)
- [Uncertainty-Aware AB3DMOT by Variational 3D Object Detection](https://mlsanity.com/paper/2302.05923/) (2023)
- [Uncertainty Estimation in Contrast-Enhanced MR Image Translation with Multi-Axis Fusion](https://mlsanity.com/paper/2311.12153/) (2023)
- [Bayesian Deep Learning Via Expectation Maximization and Turbo Deep Approximate Message Passing](https://mlsanity.com/paper/2402.07366/) (2024)
- [Probabilistic Neural Networks (PNNs) for Modeling Aleatoric Uncertainty in Scientific Machine Learning](https://mlsanity.com/paper/2402.13945/) (2024)
- [Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies](https://mlsanity.com/paper/2507.03007/) (2025)
- [Probabilistic Consistency in Machine Learning and Its Connection to Uncertainty Quantification](https://mlsanity.com/paper/2507.21670/) (2025)
- [Generalized Evidential Deep Learning: From a Bayesian Perspective](https://mlsanity.com/paper/2605.25599/) (2026)
- [Variational Voxel Pseudo Image Tracking](https://mlsanity.com/paper/2302.05914/) (2023)
- [Gibbs Sampling the Posterior of Neural Networks](https://mlsanity.com/paper/2306.02729/) (2023)
- [Bayesian Reasoning for Physics Informed Neural Networks](https://mlsanity.com/paper/2308.13222/) (2023)
- [Variational Inference for GARCH-family Models](https://mlsanity.com/paper/2310.03435/) (2023)
- [A Priori Uncertainty Quantification of Reacting Turbulence Closure Models using Bayesian Neural Networks](https://mlsanity.com/paper/2402.18729/) (2024)
- [BayesJudge: Bayesian Kernel Language Modelling with Confidence Uncertainty in Legal Judgment Prediction](https://mlsanity.com/paper/2404.10481/) (2024)
- [Robust Domain Generalisation with Causal Invariant Bayesian Neural Networks](https://mlsanity.com/paper/2410.06349/) (2024)
- [Multi-Attribute Multi-Grained Adaptation of Pre-Trained Language Models for Text Understanding from Bayesian Perspective](https://mlsanity.com/paper/2503.06085/) (2025)
- [Variational Graph Convolutional Neural Networks](https://mlsanity.com/paper/2507.01699/) (2025)
- [Robust Classification of Oral Cancer with Limited Training Data](https://mlsanity.com/paper/2510.01547/) (2025)
- [Bayesian Inference of Primordial Magnetic Field Parameters from CMB with Spherical Graph Neural Networks](https://mlsanity.com/paper/2510.20795/) (2025)
- [Bayes-DIC Net: Estimating Digital Image Correlation Uncertainty with Bayesian Neural Networks](https://mlsanity.com/paper/2512.04323/) (2025)
- [Cross-Dataset Transfer and Unknown-Class Detection in Imbalanced SAR Ship Classification](https://mlsanity.com/paper/2609.31658/) (2026)

## Abstract (arXiv)

The last decade witnessed a growing interest in Bayesian learning. Yet, the technicality of the topic and the multitude of ingredients involved therein, besides the complexity of turning theory into practical implementations, limit the use of the Bayesian learning paradigm, preventing its widespread adoption across different fields and applications. This self-contained survey engages and introduces readers to the principles and algorithms of Bayesian Learning for Neural Networks. It provides an introduction to the topic from an accessible, practical-algorithmic perspective. Upon providing a general introduction to Bayesian Neural Networks, we discuss and present both standard and recent approaches for Bayesian inference, with an emphasis on solutions relying on Variational Inference and the use of Natural gradients. We also discuss the use of manifold optimization as a state-of-the-art approach to Bayesian learning. We examine the characteristic properties of all the discussed methods, and provide pseudo-codes for their implementation, paying attention to practical aspects, such as the computation of the gradients.