mlsanityData to 14 Oct 2026

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations

arXiv 1906.02914 ↗First posted 7 Jun 2019stat.ML cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to minimal exponential-family (EF) approximations. In this paper, we extend their application to estimate structured approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a minimal conditional-EF representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods with reparameterization gradients. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 3; 2023: 14; 2024: 10; 2025: 9; 2026: 7.

51 papers.

Sort:

  1. 2021The Bayesian Learning Rulecited by 99
  2. 2022Variational inference via Wasserstein gradient flowscited by 85
  3. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  4. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  5. 2021Challenges and Opportunities in High-dimensional Variational Inferencecited by 35
  6. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  7. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  8. 2022A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Modelscited by 22
  9. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  10. 2024Outlier-robust Kalman Filtering through Generalised Bayescited by 21
  11. 2023Sampling via Gradient Flows in the Space of Probability Measurescited by 20
  12. 2023Variational Inference with Gaussian Score Matchingcited by 17
38 more
  1. 2023Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariancecited by 16
  2. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 16
  3. 2023Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descentcited by 13
  4. 2024Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalitiescited by 13
  5. 2023Variational Gaussian Process Diffusion Processescited by 11
  6. 2024Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussianscited by 11
  7. 2021A practical tutorial on Variational Bayescited by 10
  8. 2024Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flowscited by 9
  9. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8
  10. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  11. 2021Meta-strategy for Learning Tuning Parameters with Guaranteescited by 6
  12. 2025Variational Inference with Mixtures of Isotropic Gaussianscited by 6
  13. 2023Wasserstein Gradient Flow over Variational Parameter Space for Variational Inferencecited by 5
  14. 2025SING: SDE Inference via Natural Gradientscited by 5
  15. 2024Provably Scalable Black-Box Variational Inference with Structured Variational Familiescited by 4
  16. 2023BALANCE: Bayesian Linear Attribution for Root Cause Localizationcited by 3
  17. 2023The Lie-Group Bayesian Learning Rulecited by 3
  18. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  19. 2025Natural Gradient VI: Guarantees for Non-Conjugate Modelscited by 3
  20. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 3
  21. 2025Information Geometry of Variational Bayescited by 2
  22. 2021Stein Latent Optimization for Generative Adversarial Networkscited by 1
  23. 2021Natural Gradient Variational Inference with Gaussian Mixture Modelscited by 1
  24. 2022VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognitioncited by 1
  25. 2023Variational Bayes Made Easycited by 1
  26. 2023Optimising Distributions with Natural Gradient Surrogatescited by 1
  27. 2023Joint State Estimation and Noise Identification Based on Variational Optimizationcited by 1
  28. 2024Variational Model Merging for Pareto Front Estimation in Multitask Finetuningcited by 1
  29. 2025Rotated Mean-Field Variational Inference and Iterative Gaussianizationcited by 1
  30. 2026Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedulescited by 1
  31. 2021Boosting Variational Inference With Locally Adaptive Step-Sizescited by 0
  32. 2023Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Modelscited by 0
  33. 2024Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradientcited by 0
  34. 2025Spectral-factorized Positive-definite Curvature Learning for NN Trainingcited by 0
  35. 2025Geometric Convergence Analysis of Variational Inference via Bregman Divergencescited by 0
  36. 2026Learning Mixture Density via Natural Gradient Expectation Maximizationcited by 0
  37. 2026Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Spacecited by 0
  38. 2026Large-scale empirical tuning and comparison of default optimizers for variational inferencecited by 0

50 most cited of 51 All 51 (JSON)