mlsanityData to 14 Oct 2026

Optimization Guarantees for Square-Root Natural-Gradient Variational Inference

arXiv 2507.07853 ↗First posted 10 Jul 2025cs.LG cs.AI stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 75 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

41 of 75 references.

References · 41

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2015Weight Uncertainty in Neural Networkscited by 554
  4. 2013Gaussian Processes for Big Datacited by 383
  5. 2013Black Box Variational Inferencecited by 319
  6. 2014Scalable Variational Gaussian Process Classificationcited by 228
  7. 2019Practical Deep Learning with Bayesian Principlescited by 158
  8. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  9. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  10. 2016Composing graphical models with neural networks for structured representations and fast inferencecited by 113
  11. 2021The Bayesian Learning Rulecited by 99
  12. 2022Variational inference via Wasserstein gradient flowscited by 85
29 more
  1. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  2. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  3. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  4. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  5. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 38
  6. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  7. 2015On Graduated Optimization for Stochastic Non-Convex Problemscited by 34
  8. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  9. 2023Provable convergence guarantees for black-box variational inferencecited by 29
  10. 2019Convergence Rates of Variational Inference in Sparse Deep Learningcited by 27
  11. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  12. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  13. 2024Batch and match: black-box variational inference with a score-based divergencecited by 22
  14. 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
  15. 2015Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functionscited by 19
  16. 2021Dual Parameterization of Sparse Variational Gaussian Processescited by 19
  17. 2019A Generalization Bound for Online Variational Inferencecited by 17
  18. 2023Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariancecited by 16
  19. 2023Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descentcited by 13
  20. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  21. 2015A trust-region method for stochastic variational inference with applications to streaming datacited by 9
  22. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  23. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8
  24. 2017Fast Black-box Variational Inference through Stochastic Trust-Region Optimizationcited by 7
  25. 2017Variational Adaptive-Newton Method for Explorative Learningcited by 7
  26. 2021Structured second-order methods via natural gradient descentcited by 7
  27. 2019The Variational Predictive Natural Gradientcited by 4
  28. 2024Provably Scalable Black-Box Variational Inference with Structured Variational Familiescited by 4
  29. 2023Natural Gradient Variational Bayes without Fisher Matrix Analytic Calculation and Its Inversioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedulescited by 1
  2. 2026Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Spacecited by 0
  3. 2026Large-scale empirical tuning and comparison of default optimizers for variational inferencecited by 0