# Bayesian Neural Networks: Essentials

arXiv: https://arxiv.org/abs/2106.13594 · HTML: https://mlsanity.com/paper/2106.13594/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)
- [Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning](https://mlsanity.com/paper/1907.00865/) (2019)
- [Deep Ensembles from a Bayesian Perspective](https://mlsanity.com/paper/2105.13283/) (2021)
- [On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation](https://mlsanity.com/paper/2001.08049/) (2020)
- [The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning](https://mlsanity.com/paper/1811.12535/) (2018)
- [Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Models](https://mlsanity.com/paper/2106.00120/) (2021)

## Built on by (most cited first, 7 of 7)

- [Generative Pretraining for Black-Box Optimization](https://mlsanity.com/paper/2206.10786/) (2022)
- [Hybrid Deep Learning Model using SPCAGAN Augmentation for Insider Threat Analysis](https://mlsanity.com/paper/2203.02855/) (2022)
- [Constraining cosmological parameters from N-body simulations with Variational Bayesian Neural Networks](https://mlsanity.com/paper/2301.03991/) (2023)
- [Neural Network Prediction of Strong Lensing Systems with Domain Adaptation and Uncertainty Quantification](https://mlsanity.com/paper/2411.03334/) (2024)
- [Constraining cosmological parameters from N-body simulations with Bayesian Neural Networks](https://mlsanity.com/paper/2112.11865/) (2021)
- [Hybrid Bayesian Neural Networks with Functional Probabilistic Layers](https://mlsanity.com/paper/2107.07014/) (2021)
- [Partial Trace-Class Bayesian Neural Networks](https://mlsanity.com/paper/2511.01628/) (2025)

## Abstract (arXiv)

Bayesian neural networks utilize probabilistic layers that capture uncertainty over weights and activations, and are trained using Bayesian inference. Since these probabilistic layers are designed to be drop-in replacement of their deterministic counter parts, Bayesian neural networks provide a direct and natural way to extend conventional deep neural networks to support probabilistic deep learning. However, it is nontrivial to understand, design and train Bayesian neural networks due to their complexities. We discuss the essentials of Bayesian neural networks including duality (deep neural networks, probabilistic models), approximate Bayesian inference, Bayesian priors, Bayesian posteriors, and deep variational learning. We use TensorFlow Probability APIs and code examples for illustration. The main problem with Bayesian neural networks is that the architecture of deep neural networks makes it quite redundant, and costly, to account for uncertainty for a large number of successive layers. Hybrid Bayesian neural networks, which use few probabilistic layers judicially positioned in the networks, provide a practical solution.