mlsanityData to 7 Oct 2026

Understanding Stochastic Natural Gradient Variational Inference

arXiv 2406.01870 ↗First posted 4 Jun 2024cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Stochastic natural gradient variational inference (NGVI) is a popular posterior inference method with applications in various probabilistic models. Despite its wide usage, little is known about the non-asymptotic convergence rate in the stochastic setting. We aim to lessen this gap and provide a better understanding. For conjugate likelihoods, we prove the first O(frac1T) non-asymptotic convergence rate of stochastic NGVI. The complexity is no worse than stochastic gradient descent (aka black-box variational inference) and the rate likely has better constant dependency that leads to faster convergence in practice. For non-conjugate likelihoods, we show that stochastic NGVI with the canonical parameterization implicitly optimizes a non-convex objective. Thus, a global convergence rate of O(frac1T) is unlikely without some significant new understanding of optimizing the ELBO using natural gradients.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 60 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

34 of 60 references.

References · 34

  1. 2016Variational Inference: A Review for Statisticianscited by 1,466
  2. 2014Convex Optimization: Algorithms and Complexitycited by 569
  3. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  4. 2014New insights and perspectives on the natural gradient methodcited by 435
  5. 2013Gaussian Processes for Big Datacited by 377
  6. 2013Black Box Variational Inferencecited by 317
  7. 2019A Theory of Regularized Markov Decision Processescited by 279
  8. 2014Scalable Variational Gaussian Process Classificationcited by 226
  9. 2019Practical Deep Learning with Bayesian Principlescited by 158
  10. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  11. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
  12. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
22 more
  1. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  2. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  3. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  4. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  5. 2013The Information Geometry of Mirror Descentcited by 53
  6. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  7. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  8. 2023High Probability Convergence of Stochastic Gradient Methodscited by 48
  9. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  10. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  11. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  12. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  13. 2023Provable convergence guarantees for black-box variational inferencecited by 29
  14. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  15. 2022A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Modelscited by 22
  16. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  17. 2015Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functionscited by 19
  18. 2018Accelerating Natural Gradient with Higher-Order Invariancecited by 17
  19. 2019A Generalization Bound for Online Variational Inferencecited by 17
  20. 2018Fastest Rates for Stochastic Mirror Descent Methodscited by 14
  21. 2018Optimal transport natural gradient for statistical manifolds with continuous sample spacecited by 14
  22. 2024Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergencecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  2. 2025Natural Gradient VI: Guarantees for Non-Conjugate Modelscited by 3
  3. 2026Scalable Mean-Field Variational Inference via Preconditioned Primal-Dual Optimizationcited by 1
  4. 2026Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedulescited by 1
  5. 2025ONG: Orthogonal Natural Gradient Descentcited by 0
  6. 2026A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regressioncited by 0
  7. 2026Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Spacecited by 0
  8. 2026SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batchingcited by 0