# Online Natural Gradient as a Kalman Filter

arXiv: https://arxiv.org/abs/1703.00209 · HTML: https://mlsanity.com/paper/1703.00209/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2017.

## Built on by (most cited first, 25 of 25)

- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Outlier-robust Kalman Filtering through Generalised Bayes](https://mlsanity.com/paper/2405.05646/) (2024)
- [Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance](https://mlsanity.com/paper/2302.11024/) (2023)
- [Efficient Online Bayesian Inference for Neural Bandits](https://mlsanity.com/paper/2112.00195/) (2021)
- [Bayesian Online Natural Gradient (BONG)](https://mlsanity.com/paper/2405.19681/) (2024)
- [Scalable Bayesian Learning with posteriors](https://mlsanity.com/paper/2406.00104/) (2024)
- [Gaussian Pre-Activations in Neural Networks: Myth or Reality?](https://mlsanity.com/paper/2205.12379/) (2022)
- [EVCL: Elastic Variational Continual Learning with Weight Consolidation](https://mlsanity.com/paper/2406.15972/) (2024)
- [A unifying framework for generalised Bayesian online learning in non-stationary environments](https://mlsanity.com/paper/2411.10153/) (2024)
- [On Feynman--Kac training of partial Bayesian neural networks](https://mlsanity.com/paper/2310.19608/) (2023)
- [Implicit Maximum a Posteriori Filtering via Adaptive Optimization](https://mlsanity.com/paper/2311.10580/) (2023)
- [Martingale Posterior Neural Networks for Fast Sequential Decision Making](https://mlsanity.com/paper/2506.11898/) (2025)
- [Detecting Toxic Flow](https://mlsanity.com/paper/2312.05827/) (2023)
- [Predictive coding, precision and natural gradients](https://mlsanity.com/paper/2111.06942/) (2021)
- [A State-Space Perspective on Modelling and Inference for Online Skill Rating](https://mlsanity.com/paper/2308.02414/) (2023)
- [Information Geometry for the Working Information Theorist](https://mlsanity.com/paper/2310.03884/) (2023)
- [Adaptive time series forecasting with markovian variance switching](https://mlsanity.com/paper/2402.14684/) (2024)
- [LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models](https://mlsanity.com/paper/2410.11551/) (2024)
- [Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering](https://mlsanity.com/paper/2511.01694/) (2025)
- [Gradient Regularized Natural Gradients](https://mlsanity.com/paper/2601.18420/) (2026)
- [Training Algorithm Matters for the Performance of Neural Network Potential: A Case Study of Adam and the Kalman Filter Optimizers](https://mlsanity.com/paper/2109.03769/) (2021)
- [Online Continual Learning for Time Series: a Natural Score-driven Approach](https://mlsanity.com/paper/2601.12931/) (2026)
- [Generalized Information Gathering Under Dynamics Uncertainty](https://mlsanity.com/paper/2601.21988/) (2026)

## Abstract (arXiv)

We cast Amari's natural gradient in statistical learning as a specific case of Kalman filtering. Namely, applying an extended Kalman filter to estimate a fixed unknown parameter of a probabilistic model from a series of observations, is rigorously equivalent to estimating this parameter via an online stochastic natural gradient descent on the log-likelihood of the observations. In the i.i.d. case, this relation is a consequence of the "information filter" phrasing of the extended Kalman filter. In the recurrent (state space, non-i.i.d.) case, we prove that the joint Kalman filter over states and parameters is a natural gradient on top of real-time recurrent learning (RTRL), a classical algorithm to train recurrent models. This exact algebraic correspondence provides relevant interpretations for natural gradient hyperparameters such as learning rates or initialization and regularization of the Fisher information matrix.