# Kalman Bayesian Neural Networks for Closed-form Online Learning

arXiv: https://arxiv.org/abs/2110.00944 · HTML: https://mlsanity.com/paper/2110.00944/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 3 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation](https://mlsanity.com/paper/2006.07584/) (2020)
- [Bayesian Perceptron: Towards fully Bayesian Neural Networks](https://mlsanity.com/paper/2009.01730/) (2020)

## Built on by (most cited first, 9 of 9)

- [A Comprehensive Review of Digital Twin -- Part 1: Modeling and Twinning Enabling Technologies](https://mlsanity.com/paper/2208.14197/) (2022)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Particle-based Online Bayesian Sampling](https://mlsanity.com/paper/2302.14796/) (2023)
- [Uncertainty-driven Exploration Strategies for Online Grasp Learning](https://mlsanity.com/paper/2309.12038/) (2023)
- [Assumed Density Filtering and Smoothing with Neural Network Surrogate Models](https://mlsanity.com/paper/2511.09016/) (2025)
- [Staying Alive: Online Neural Network Maintenance and Systemic Drift](https://mlsanity.com/paper/2503.17681/) (2025)
- [Kalman Bayesian Transformer](https://mlsanity.com/paper/2509.10695/) (2025)
- [From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network](https://mlsanity.com/paper/2512.13111/) (2025)
- [Full-Covariance Smoothing of Bayesian Neural Networks for Online Adaptation](https://mlsanity.com/paper/2609.27244/) (2026)

## Abstract (arXiv)

Compared to point estimates calculated by standard neural networks, Bayesian neural networks (BNN) provide probability distributions over the output predictions and model parameters, i.e., the weights. Training the weight distribution of a BNN, however, is more involved due to the intractability of the underlying Bayesian inference problem and thus, requires efficient approximations. In this paper, we propose a novel approach for BNN learning via closed-form Bayesian inference. For this purpose, the calculation of the predictive distribution of the output and the update of the weight distribution are treated as Bayesian filtering and smoothing problems, where the weights are modeled as Gaussian random variables. This allows closed-form expressions for training the network's parameters in a sequential/online fashion without gradient descent. We demonstrate our method on several UCI datasets and compare it to the state of the art.