# A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models

arXiv: https://arxiv.org/abs/2209.11533 · HTML: https://mlsanity.com/paper/2209.11533/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations](https://mlsanity.com/paper/1906.02914/) (2019)
- [Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models](https://mlsanity.com/paper/1803.09151/) (2018)
- [Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models](https://mlsanity.com/paper/1807.04489/) (2018)
- [Variational Boosting: Iteratively Refining Posterior Approximations](https://mlsanity.com/paper/1611.06585/) (2016)
- [Handling the Positive-Definite Constraint in the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10060/) (2020)
- [Differentiable Trust Region Layers for Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09207/) (2021)
- [Boosting Variational Inference](https://mlsanity.com/paper/1611.05559/) (2016)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions](https://mlsanity.com/paper/1511.00146/) (2015)
- [Automatic Differentiation Variational Inference with Mixtures](https://mlsanity.com/paper/2003.01687/) (2020)
- [Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures](https://mlsanity.com/paper/1910.13398/) (2019)
- [Trust-Region Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/1907.04710/) (2019)
- [Compatible Natural Gradient Policy Search](https://mlsanity.com/paper/1902.02823/) (2019)
- [A trust-region method for stochastic variational inference with applications to streaming data](https://mlsanity.com/paper/1505.07649/) (2015)
- [Expected Information Maximization: Using the I-Projection for Mixture Density Estimation](https://mlsanity.com/paper/2001.08682/) (2020)
- [Fast Black-box Variational Inference through Stochastic Trust-Region Optimization](https://mlsanity.com/paper/1706.02375/) (2017)
- [Monotonic Alpha-divergence Minimisation for Variational Inference](https://mlsanity.com/paper/2103.05684/) (2021)
- [Assisted Teleoperation in Changing Environments with a Mixture of Virtual Guides](https://mlsanity.com/paper/2008.05251/) (2020)

## Built on by (most cited first, 22 of 22)

- [NETS: A Non-Equilibrium Transport Sampler](https://mlsanity.com/paper/2410.02711/) (2024)
- [Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling](https://mlsanity.com/paper/2406.07423/) (2024)
- [Underdamped Diffusion Bridges with Applications to Sampling](https://mlsanity.com/paper/2503.01006/) (2025)
- [Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference](https://mlsanity.com/paper/2508.12511/) (2025)
- [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023)
- [Variational Distillation of Diffusion Policies into Mixture of Experts](https://mlsanity.com/paper/2406.12538/) (2024)
- [Amortized In-Context Bayesian Posterior Estimation](https://mlsanity.com/paper/2502.06601/) (2025)
- [Learned Reference-based Diffusion Sampling for multi-modal distributions](https://mlsanity.com/paper/2410.19449/) (2024)
- [Learning Boltzmann Generators via Constrained Mass Transport](https://mlsanity.com/paper/2510.18460/) (2025)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Variational Inference with Mixtures of Isotropic Gaussians](https://mlsanity.com/paper/2506.13613/) (2025)
- [End-To-End Learning of Gaussian Mixture Priors for Diffusion Sampler](https://mlsanity.com/paper/2503.00524/) (2025)
- [An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control](https://mlsanity.com/paper/2410.05163/) (2024)
- [Model-Informed Flows for Bayesian Inference](https://mlsanity.com/paper/2505.24243/) (2025)
- [Natural Gradient VI: Guarantees for Non-Conjugate Models](https://mlsanity.com/paper/2510.19163/) (2025)
- [Learning Multimodal Latent Dynamics for Human-Robot Interaction](https://mlsanity.com/paper/2311.16380/) (2023)
- [Even More Guarantees for Variational Inference in the Presence of Symmetries](https://mlsanity.com/paper/2604.21407/) (2026)
- [Information-Theoretic Trust Regions for Stochastic Gradient-Based Optimization](https://mlsanity.com/paper/2310.20574/) (2023)
- [Latent Task-Specific Graph Network Simulators](https://mlsanity.com/paper/2311.05256/) (2023)
- [Learning Mixture Density via Natural Gradient Expectation Maximization](https://mlsanity.com/paper/2602.10602/) (2026)
- [Continuous Variational Synthesis](https://mlsanity.com/paper/2609.35083/) (2026)
- [Adaptive mixture variational inference for spike-and-slab regression](https://mlsanity.com/paper/2609.38656/) (2026)

## Abstract (arXiv)

Variational inference with Gaussian mixture models (GMMs) enables learning of highly tractable yet multi-modal approximations of intractable target distributions with up to a few hundred dimensions. The two currently most effective methods for GMM-based variational inference, VIPS and iBayes-GMM, both employ independent natural gradient updates for the individual components and their weights. We show for the first time, that their derived updates are equivalent, although their practical implementations and theoretical guarantees differ. We identify several design choices that distinguish both approaches, namely with respect to sample selection, natural gradient estimation, stepsize adaptation, and whether trust regions are enforced or the number of components adapted. We argue that for both approaches, the quality of the learned approximations can heavily suffer from the respective design choices: By updating the individual components using samples from the mixture model, iBayes-GMM often fails to produce meaningful updates to low-weight components, and by using a zero-order method for estimating the natural gradient, VIPS scales badly to higher-dimensional problems. Furthermore, we show that information-geometric trust-regions (used by VIPS) are effective even when using first-order natural gradient estimates, and often outperform the improved Bayesian learning rule (iBLR) update used by iBayes-GMM. We systematically evaluate the effects of design choices and show that a hybrid approach significantly outperforms both prior works. Along with this work, we publish our highly modular and efficient implementation for natural gradient variational inference with Gaussian mixture models, which supports 432 different combinations of design choices, facilitates the reproduction of all our experiments, and may prove valuable for the practitioner.