# Bayesian Online Natural Gradient (BONG)

arXiv: https://arxiv.org/abs/2405.19681 · HTML: https://mlsanity.com/paper/2405.19681/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 19 of 19)

- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient](https://mlsanity.com/paper/1811.04504/) (2018)
- [The Many Faces of Exponential Weights in Online Learning](https://mlsanity.com/paper/1802.07543/) (2018)
- [Online Natural Gradient as a Kalman Filter](https://mlsanity.com/paper/1703.00209/) (2017)
- [Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective](https://mlsanity.com/paper/2402.03496/) (2024)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Outlier-robust Kalman Filtering through Generalised Bayes](https://mlsanity.com/paper/2405.05646/) (2024)
- [A Generalization Bound for Online Variational Inference](https://mlsanity.com/paper/1904.03920/) (2019)
- [Kalman Filter for Online Classification of Non-Stationary Data](https://mlsanity.com/paper/2306.08448/) (2023)
- [The limited-memory recursive variational Gaussian approximation (L-RVGA)](https://mlsanity.com/paper/2303.14195/) (2023)
- [Implicit Maximum a Posteriori Filtering via Adaptive Optimization](https://mlsanity.com/paper/2311.10580/) (2023)
- [Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation](https://mlsanity.com/paper/2404.12766/) (2024)

## Built on by (most cited first, 14 of 14)

- [Knowledge Adaptation as Posterior Correction](https://mlsanity.com/paper/2506.14262/) (2025)
- [A unifying framework for generalised Bayesian online learning in non-stationary environments](https://mlsanity.com/paper/2411.10153/) (2024)
- [SING: SDE Inference via Natural Gradients](https://mlsanity.com/paper/2506.17796/) (2025)
- [Martingale Posterior Neural Networks for Fast Sequential Decision Making](https://mlsanity.com/paper/2506.11898/) (2025)
- [Natural Gradient VI: Guarantees for Non-Conjugate Models](https://mlsanity.com/paper/2510.19163/) (2025)
- [SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks](https://mlsanity.com/paper/2606.23357/) (2026)
- [LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models](https://mlsanity.com/paper/2410.11551/) (2024)
- [Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering](https://mlsanity.com/paper/2511.01694/) (2025)
- [Gradient Regularized Natural Gradients](https://mlsanity.com/paper/2601.18420/) (2026)
- [Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes](https://mlsanity.com/paper/2505.15638/) (2025)
- [Variational predictive resampling](https://mlsanity.com/paper/2605.11168/) (2026)
- [A probabilistic framework for online test-time adaptation](https://mlsanity.com/paper/2606.26457/) (2026)
- [Fast rates in Bayesian online learning with approximate posteriors](https://mlsanity.com/paper/2608.25706/) (2026)
- [Online Learning via Learned Latent Bayesian Tracking](https://mlsanity.com/paper/2609.31559/) (2026)

## Abstract (arXiv)

We propose a novel approach to sequential Bayesian inference based on variational Bayes (VB). The key insight is that, in the online setting, we do not need to add the KL term to regularize to the prior (which comes from the posterior at the previous timestep); instead we can optimize just the expected log-likelihood, performing a single step of natural gradient descent starting at the prior predictive. We prove this method recovers exact Bayesian inference if the model is conjugate. We also show how to compute an efficient deterministic approximation to the VB objective, as well as our simplified objective, when the variational distribution is Gaussian or a sub-family, including the case of a diagonal plus low-rank precision matrix. We show empirically that our method outperforms other online VB methods in the non-conjugate setting, such as online learning for neural networks, especially when controlling for computational costs.