mlsanityData to 14 Oct 2026

Provable convergence guarantees for black-box variational inference

arXiv 2306.03638 ↗First posted 4 Jun 2023cs.LG math.OC stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in existing stochastic optimization proofs: namely the challenge of gradient estimators with unusual noise bounds, and a composite non-smooth objective. For dense Gaussian variational families, we observe that existing gradient estimators based on reparameterization satisfy a quadratic noise bound and give novel convergence guarantees for proximal and projected stochastic gradient descent using this bound. This provides rigorous guarantees that methods similar to those used in practice converge on realistic inference problems.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

36 of 62 references.

References · 36

  1. 2016Variational Inference: A Review for Statisticianscited by 1,472
  2. 2018Computational Optimal Transportcited by 780
  3. 2013Black Box Variational Inferencecited by 319
  4. 2017Advances in Variational Inferencecited by 236
  5. 2019SGD: General Analysis and Improved Ratescited by 207
  6. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  7. 2016Automatic Differentiation Variational Inferencecited by 159
  8. 2018Stochastic model-based minimization of weakly convex functionscited by 159
  9. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  10. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 133
  11. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 98
  12. 2022Variational inference via Wasserstein gradient flowscited by 85
24 more
  1. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  2. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 82
  3. 2017Concentration of tempered posteriors and of their variational approximationscited by 66
  4. 2018Yes, but Did It Work?: Evaluating Variational Inferencecited by 53
  5. 2018Variance reduction properties of the reparameterization trickcited by 41
  6. 2020Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimizationcited by 36
  7. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  8. 2021Challenges and Opportunities in High-dimensional Variational Inferencecited by 35
  9. 2013Automated Variational Inference in Probabilistic Programmingcited by 34
  10. 2016The Generalized Reparameterization Gradientcited by 30
  11. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  12. 2017Reducing Reparameterization Gradient Variancecited by 27
  13. 2018Quasi-Monte Carlo Variational Inferencecited by 24
  14. 2018Using Large Ensembles of Control Variates for Variational Inferencecited by 21
  15. 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
  16. 2015Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functionscited by 19
  17. 2016Gaussian variational approximation with sparse precision matricescited by 15
  18. 2019Multilevel Monte Carlo Variational Inferencecited by 15
  19. 2020Approximation Based Variance Reduction for Reparameterization Gradientscited by 10
  20. 2022A Framework for Improving the Reliability of Black-box Variational Inferencecited by 9
  21. 2015Fast Second-Order Stochastic Backpropagation for Variational Inferencecited by 8
  22. 2016Overdispersed Black-Box Variational Inferencecited by 8
  23. 2021The computational asymptotics of Gaussian variational inference and the Laplace approximationcited by 8
  24. 2017Fast Black-box Variational Inference through Stochastic Trust-Region Optimizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 7; 2026: 9.

29 papers.

Sort:

By area:

  1. 2024Batch and match: black-box variational inference with a score-based divergencecited by 22
  2. 2023Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein spacecited by 20
  3. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 17
  4. 2024Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussianscited by 11
  5. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  6. 2024Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrixcited by 9
  7. 2024Kernel Semi-Implicit Variational Inferencecited by 8
  8. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8
  9. 2025Variational Inference with Mixtures of Isotropic Gaussianscited by 6
  10. 2026Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variancecited by 5
  11. 2024Provably Scalable Black-Box Variational Inference with Structured Variational Familiescited by 4
  12. 2024Efficient Mixture Learning in Black-Box Variational Inferencecited by 3
17 more
  1. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  2. 2025Natural Gradient VI: Guarantees for Non-Conjugate Modelscited by 3
  3. 2024Demystifying SGD with Doubly Stochastic Gradientscited by 2
  4. 2024Stochastic variance-reduced Gaussian variational inference on the Bures-Wasserstein manifoldcited by 2
  5. 2025Fisher meets Feynman: score-based variational inference with a product of expertscited by 2
  6. 2023Distributed Variational Inference for Online Supervised Learningcited by 1
  7. 2024Understanding and mitigating difficulties in posterior predictive evaluationcited by 1
  8. 2025Globally Convergent Variational Inferencecited by 1
  9. 2025Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inferencecited by 1
  10. 2026A Kernel Approach for Semi-implicit Variational Inferencecited by 1
  11. 2026Large-scale Score-based Variational Posterior Inference for Bayesian Deep Neural Networkscited by 1
  12. 2026Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedulescited by 1
  13. 2026Even More Guarantees for Variational Inference in the Presence of Symmetriescited by 1
  14. 2026Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Spacecited by 0
  15. 2026SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batchingcited by 0
  16. 2026Large-scale empirical tuning and comparison of default optimizers for variational inferencecited by 0
  17. 2026Ordinary Nonconvex SGD under Distance-Dependent Moments: Finite-Horizon Stationarity and Nagaev Boundscited by 0