# Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations

arXiv: https://arxiv.org/abs/1906.02914 · HTML: https://mlsanity.com/paper/1906.02914/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 30 of 51)

- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Variational inference via Wasserstein gradient flows](https://mlsanity.com/paper/2205.15902/) (2022)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space](https://mlsanity.com/paper/2304.05398/) (2023)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective](https://mlsanity.com/paper/2402.03496/) (2024)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [On the Convergence of Black-Box Variational Inference](https://mlsanity.com/paper/2305.15349/) (2023)
- [Outlier-robust Kalman Filtering through Generalised Bayes](https://mlsanity.com/paper/2405.05646/) (2024)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023)
- [Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance](https://mlsanity.com/paper/2302.11024/) (2023)
- [Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization](https://mlsanity.com/paper/2509.03378/) (2025)
- [Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent](https://mlsanity.com/paper/2305.14076/) (2023)
- [Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities](https://mlsanity.com/paper/2407.15693/) (2024)
- [Variational Gaussian Process Diffusion Processes](https://mlsanity.com/paper/2306.02066/) (2023)
- [Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians](https://mlsanity.com/paper/2406.04012/) (2024)
- [A practical tutorial on Variational Bayes](https://mlsanity.com/paper/2103.01327/) (2021)
- [Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flows](https://mlsanity.com/paper/2406.17263/) (2024)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory](https://mlsanity.com/paper/2602.09314/) (2026)
- [Meta-strategy for Learning Tuning Parameters with Guarantees](https://mlsanity.com/paper/2102.02504/) (2021)
- [Variational Inference with Mixtures of Isotropic Gaussians](https://mlsanity.com/paper/2506.13613/) (2025)
- [Wasserstein Gradient Flow over Variational Parameter Space for Variational Inference](https://mlsanity.com/paper/2310.16705/) (2023)
- [SING: SDE Inference via Natural Gradients](https://mlsanity.com/paper/2506.17796/) (2025)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [BALANCE: Bayesian Linear Attribution for Root Cause Localization](https://mlsanity.com/paper/2301.13572/) (2023)
- [The Lie-Group Bayesian Learning Rule](https://mlsanity.com/paper/2303.04397/) (2023)
- [Optimization Guarantees for Square-Root Natural-Gradient Variational Inference](https://mlsanity.com/paper/2507.07853/) (2025)

## Abstract (arXiv)

Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to minimal exponential-family (EF) approximations. In this paper, we extend their application to estimate structured approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a minimal conditional-EF representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods with reparameterization gradients. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.