# Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization

arXiv: https://arxiv.org/abs/1810.12273 · HTML: https://mlsanity.com/paper/1810.12273/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 8 of 8)

- [Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism](https://mlsanity.com/paper/2506.01260/) (2025)
- [DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction](https://mlsanity.com/paper/2410.03883/) (2024)
- [KOALA: A Kalman Optimization Algorithm with Loss Adaptivity](https://mlsanity.com/paper/2107.03331/) (2021)
- [Signal Processing Meets SGD: From Momentum to Filter](https://mlsanity.com/paper/2311.02818/) (2023)
- [Dynamic Momentum Recalibration in Online Gradient Learning](https://mlsanity.com/paper/2603.06120/) (2026)
- [FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction](https://mlsanity.com/paper/2605.03425/) (2026)
- [Variational Stochastic Gradient Descent for Deep Neural Networks](https://mlsanity.com/paper/2404.06549/) (2024)
- [VCMM: Variance-Calibrated Momentum for Multimodal Learning](https://mlsanity.com/paper/2609.27577/) (2026)

## Abstract (arXiv)

We introduce Kalman Gradient Descent, a stochastic optimization algorithm that uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent by filtering the gradient estimates. We present both a theoretical analysis of convergence in a non-convex setting and experimental results which demonstrate improved performance on a variety of machine learning areas including neural networks and black box variational inference. We also present a distributed version of our algorithm that enables large-dimensional optimization, and we extend our algorithm to SGD with momentum and RMSProp.