# Provable convergence guarantees for black-box variational inference

arXiv: https://arxiv.org/abs/2306.03638 · HTML: https://mlsanity.com/paper/2306.03638/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 30 of 36)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [SGD: General Analysis and Improved Rates](https://mlsanity.com/paper/1901.09401/) (2019)
- [Handbook of Convergence Theorems for (Stochastic) Gradient Methods](https://mlsanity.com/paper/2301.11235/) (2023)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Stochastic model-based minimization of weakly convex functions](https://mlsanity.com/paper/1803.06523/) (2018)
- [Better Theory for SGD in the Nonconvex World](https://mlsanity.com/paper/2002.03329/) (2020)
- [Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm](https://mlsanity.com/paper/1310.5715/) (2013)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [Variational inference via Wasserstein gradient flows](https://mlsanity.com/paper/2205.15902/) (2022)
- [A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent](https://mlsanity.com/paper/1905.11261/) (2019)
- [Unified Optimal Analysis of the (Stochastic) Gradient Method](https://mlsanity.com/paper/1907.04232/) (2019)
- [Concentration of tempered posteriors and of their variational approximations](https://mlsanity.com/paper/1706.09293/) (2017)
- [Yes, but Did It Work?: Evaluating Variational Inference](https://mlsanity.com/paper/1802.02538/) (2018)
- [Variance reduction properties of the reparameterization trick](https://mlsanity.com/paper/1809.10330/) (2018)
- [Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization](https://mlsanity.com/paper/2006.11573/) (2020)
- [Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space](https://mlsanity.com/paper/2304.05398/) (2023)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [Automated Variational Inference in Probabilistic Programming](https://mlsanity.com/paper/1301.1299/) (2013)
- [The Generalized Reparameterization Gradient](https://mlsanity.com/paper/1610.02287/) (2016)
- [Provable Smoothness Guarantees for Black-Box Variational Inference](https://mlsanity.com/paper/1901.08431/) (2019)
- [Reducing Reparameterization Gradient Variance](https://mlsanity.com/paper/1705.07880/) (2017)
- [Quasi-Monte Carlo Variational Inference](https://mlsanity.com/paper/1807.01604/) (2018)
- [Using Large Ensembles of Control Variates for Variational Inference](https://mlsanity.com/paper/1810.12482/) (2018)
- [Provable Gradient Variance Guarantees for Black-Box Variational Inference](https://mlsanity.com/paper/1906.08241/) (2019)
- [Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions](https://mlsanity.com/paper/1511.00146/) (2015)
- [Gaussian variational approximation with sparse precision matrices](https://mlsanity.com/paper/1605.05622/) (2016)
- [Multilevel Monte Carlo Variational Inference](https://mlsanity.com/paper/1902.00468/) (2019)

## Built on by (most cited first, 29 of 29)

- [Batch and match: black-box variational inference with a score-based divergence](https://mlsanity.com/paper/2402.14758/) (2024)
- [Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein space](https://mlsanity.com/paper/2312.02849/) (2023)
- [Towards Weaker Variance Assumptions for Stochastic Optimization](https://mlsanity.com/paper/2504.09951/) (2025)
- [Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians](https://mlsanity.com/paper/2406.04012/) (2024)
- [Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?](https://mlsanity.com/paper/2307.14642/) (2023)
- [Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix](https://mlsanity.com/paper/2410.11067/) (2024)
- [Kernel Semi-Implicit Variational Inference](https://mlsanity.com/paper/2405.18997/) (2024)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Variational Inference with Mixtures of Isotropic Gaussians](https://mlsanity.com/paper/2506.13613/) (2025)
- [Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance](https://mlsanity.com/paper/2604.16620/) (2026)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [Efficient Mixture Learning in Black-Box Variational Inference](https://mlsanity.com/paper/2406.07083/) (2024)
- [Optimization Guarantees for Square-Root Natural-Gradient Variational Inference](https://mlsanity.com/paper/2507.07853/) (2025)
- [Natural Gradient VI: Guarantees for Non-Conjugate Models](https://mlsanity.com/paper/2510.19163/) (2025)
- [Demystifying SGD with Doubly Stochastic Gradients](https://mlsanity.com/paper/2406.00920/) (2024)
- [Stochastic variance-reduced Gaussian variational inference on the Bures-Wasserstein manifold](https://mlsanity.com/paper/2410.02490/) (2024)
- [Fisher meets Feynman: score-based variational inference with a product of experts](https://mlsanity.com/paper/2510.21598/) (2025)
- [Distributed Variational Inference for Online Supervised Learning](https://mlsanity.com/paper/2309.02606/) (2023)
- [Understanding and mitigating difficulties in posterior predictive evaluation](https://mlsanity.com/paper/2405.19747/) (2024)
- [Globally Convergent Variational Inference](https://mlsanity.com/paper/2501.08201/) (2025)
- [Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference](https://mlsanity.com/paper/2505.21721/) (2025)
- [A Kernel Approach for Semi-implicit Variational Inference](https://mlsanity.com/paper/2601.12023/) (2026)
- [Large-scale Score-based Variational Posterior Inference for Bayesian Deep Neural Networks](https://mlsanity.com/paper/2602.05873/) (2026)
- [Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedules](https://mlsanity.com/paper/2604.00683/) (2026)
- [Even More Guarantees for Variational Inference in the Presence of Symmetries](https://mlsanity.com/paper/2604.21407/) (2026)
- [Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Space](https://mlsanity.com/paper/2602.18718/) (2026)
- [SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching](https://mlsanity.com/paper/2605.07531/) (2026)
- [Large-scale empirical tuning and comparison of default optimizers for variational inference](https://mlsanity.com/paper/2606.07841/) (2026)
- [Ordinary Nonconvex SGD under Distance-Dependent Moments: Finite-Horizon Stationarity and Nagaev Bounds](https://mlsanity.com/paper/2609.30499/) (2026)

## Abstract (arXiv)

Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in existing stochastic optimization proofs: namely the challenge of gradient estimators with unusual noise bounds, and a composite non-smooth objective. For dense Gaussian variational families, we observe that existing gradient estimators based on reparameterization satisfy a quadratic noise bound and give novel convergence guarantees for proximal and projected stochastic gradient descent using this bound. This provides rigorous guarantees that methods similar to those used in practice converge on realistic inference problems.