# Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models

arXiv: https://arxiv.org/abs/1807.04489 · HTML: https://mlsanity.com/paper/1807.04489/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2018.

## Built on by (most cited first, 30 of 38)

- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Bayesian Learning for Neural Networks: an algorithmic survey](https://mlsanity.com/paper/2211.11865/) (2022)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [Dual Parameterization of Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2111.03412/) (2021)
- [Sampling with Mirrored Stein Operators](https://mlsanity.com/paper/2106.12506/) (2021)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [Partitioned Variational Inference: A Framework for Probabilistic Federated Learning](https://mlsanity.com/paper/2202.12275/) (2022)
- [Variational Gaussian Process Diffusion Processes](https://mlsanity.com/paper/2306.02066/) (2023)
- [Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC](https://mlsanity.com/paper/2312.05705/) (2023)
- [Combining Pseudo-Point and State Space Approximations for Sum-Separable Gaussian Processes](https://mlsanity.com/paper/2106.10210/) (2021)
- [Hellinger-Kantorovich Gradient Flows: Global Exponential Decay of Entropy Functionals](https://mlsanity.com/paper/2501.17049/) (2025)
- [Knowledge Adaptation as Posterior Correction](https://mlsanity.com/paper/2506.14262/) (2025)
- [Bayesian Bilinear Neural Network for Predicting the Mid-price Dynamics in Limit-Order Book Markets](https://mlsanity.com/paper/2203.03613/) (2022)
- [Linear Time GPs for Inferring Latent Trajectories from Neural Spike Trains](https://mlsanity.com/paper/2306.01802/) (2023)
- [Wasserstein Gradient Flow over Variational Parameter Space for Variational Inference](https://mlsanity.com/paper/2310.16705/) (2023)
- [SING: SDE Inference via Natural Gradients](https://mlsanity.com/paper/2506.17796/) (2025)
- [Variational Nested Dropout](https://mlsanity.com/paper/2101.11353/) (2021)
- [Analysis of Kernel Mirror Prox for Measure Optimization](https://mlsanity.com/paper/2403.00147/) (2024)
- [Connecting Federated ADMM to Bayes](https://mlsanity.com/paper/2501.17325/) (2025)
- [Improved Stochastic Optimization of LogSumExp](https://mlsanity.com/paper/2509.24894/) (2025)
- [Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learning](https://mlsanity.com/paper/2205.11568/) (2022)
- [Federated ADMM from Bayesian Duality](https://mlsanity.com/paper/2506.13150/) (2025)
- [Optimization Guarantees for Square-Root Natural-Gradient Variational Inference](https://mlsanity.com/paper/2507.07853/) (2025)
- [DPVIm: Differentially Private Variational Inference Improved](https://mlsanity.com/paper/2210.15961/) (2022)
- [Unbiased Learning of Deep Generative Models with Structured Discrete Representations](https://mlsanity.com/paper/2306.08230/) (2023)
- [A Unifying Variational Framework for Gaussian Process Motion Planning](https://mlsanity.com/paper/2309.00854/) (2023)
- [Information Geometry of Variational Bayes](https://mlsanity.com/paper/2509.15641/) (2025)
- [Manifold Gaussian Variational Bayes on the Precision Matrix](https://mlsanity.com/paper/2210.14598/) (2022)

## Abstract (arXiv)

Bayesian inference plays an important role in advancing machine learning, but faces computational challenges when applied to complex models such as deep neural networks. Variational inference circumvents these challenges by formulating Bayesian inference as an optimization problem and solving it using gradient-based optimization. In this paper, we argue in favor of natural-gradient approaches which, unlike their gradient-based counterparts, can improve convergence by exploiting the information geometry of the solutions. We show how to derive fast yet simple natural-gradient updates by using a duality associated with exponential-family distributions. An attractive feature of these methods is that, by using natural-gradients, they are able to extract accurate local approximations for individual model components. We summarize recent results for Bayesian deep learning showing the superiority of natural-gradient approaches over their gradient counterparts.