mlsanityData to 14 Oct 2026

On the Convergence of Black-Box Variational Inference

arXiv 2305.15349 ↗First posted 24 May 2023cs.LG eess.SP math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

We provide the first convergence guarantee for full black-box variational inference (BBVI), also known as Monte Carlo variational inference. While preliminary investigations worked on simplified versions of BBVI (e.g., bounded domain, bounded support, only optimizing for the scale, and such), our setup does not need any such algorithmic modifications. Our results hold for log-smooth posterior densities with and without strong log-concavity and the location-scale variational family. Also, our analysis reveals that certain algorithm design choices commonly employed in practice, particularly, nonlinear parameterizations of the scale of the variational approximation, can result in suboptimal convergence rates. Fortunately, running BBVI with proximal stochastic gradient descent fixes these limitations, and thus achieves the strongest known convergence rate guarantees. We evaluate this theoretical insight by comparing proximal SGD against other standard implementations of BBVI on large-scale Bayesian inference problems.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 52 of its 86 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

52 of 86 references.

References · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  5. 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
  6. 2013Black Box Variational Inferencecited by 319
  7. 2017Advances in Variational Inferencecited by 236
  8. 2019SGD: General Analysis and Improved Ratescited by 207
  9. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  10. 2016Automatic Differentiation Variational Inferencecited by 159
  11. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  12. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
38 more
  1. 2018Log-concave sampling: Metropolis-Hastings algorithms are fastcited by 118
  2. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  3. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  4. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  5. 2017TensorFlow Distributionscited by 106
  6. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 98
  7. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  8. 2022Variational inference via Wasserstein gradient flowscited by 85
  9. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  10. 2018Random Shuffling Beats SGD after Finite Epochscited by 67
  11. 2017Concentration of tempered posteriors and of their variational approximationscited by 66
  12. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  13. 2019SGD without Replacement: Sharper Rates for General Smooth Convex Functionscited by 61
  14. 2020SGD with shuffling: optimal rates without component convexity and large epoch requirementscited by 58
  15. 2018Yes, but Did It Work?: Evaluating Variational Inferencecited by 53
  16. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  17. 2020Markovian Score Climbing: Variational Inference with KL(p||q)cited by 46
  18. 2018Variance reduction properties of the reparameterization trickcited by 41
  19. 2017Covariances, Robustness, and Variational Bayescited by 40
  20. 2015Black-box α-divergence Minimizationcited by 37
  21. 2020Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimizationcited by 37
  22. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  23. 2021Challenges and Opportunities in High-dimensional Variational Inferencecited by 35
  24. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  25. 2018Quasi-Monte Carlo Variational Inferencecited by 24
  26. 2019Variational Bayes under Model Misspecificationcited by 22
  27. 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
  28. 2020Robust, Accurate Stochastic Optimization for Variational Inferencecited by 21
  29. 2015Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functionscited by 19
  30. 2021Amortized Variational Inference for Simple Hierarchical Modelscited by 19
  31. 2023Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Boxcited by 17
  32. 2019Multilevel Monte Carlo Variational Inferencecited by 15
  33. 2021When is the Convergence Time of Langevin Algorithms Dimension Independent? A Composite Optimization Viewpointcited by 12
  34. 2015Fast Second-Order Stochastic Backpropagation for Variational Inferencecited by 8
  35. 2021The computational asymptotics of Gaussian variational inference and the Laplace approximationcited by 8
  36. 2017Fast Black-box Variational Inference through Stochastic Trust-Region Optimizationcited by 7
  37. 2017Proximity Variational Inferencecited by 6
  38. 2022Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradientscited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 5; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024Batch and match: black-box variational inference with a score-based divergencecited by 22
  2. 2023Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein spacecited by 20
  3. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  4. 2024Kernel Semi-Implicit Variational Inferencecited by 8
  5. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8
  6. 2024Provably Scalable Black-Box Variational Inference with Structured Variational Familiescited by 4
  7. 2024Efficient Mixture Learning in Black-Box Variational Inferencecited by 3
  8. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  9. 2025Natural Gradient VI: Guarantees for Non-Conjugate Modelscited by 3
  10. 2024Evaluating Bayesian deep learning for radio galaxy classificationcited by 2
  11. 2024Demystifying SGD with Doubly Stochastic Gradientscited by 2
  12. 2025Fisher meets Feynman: score-based variational inference with a product of expertscited by 2
10 more
  1. 2024Understanding and mitigating difficulties in posterior predictive evaluationcited by 1
  2. 2025Globally Convergent Variational Inferencecited by 1
  3. 2025Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inferencecited by 1
  4. 2026A Kernel Approach for Semi-implicit Variational Inferencecited by 1
  5. 2026Large-scale Score-based Variational Posterior Inference for Bayesian Deep Neural Networkscited by 1
  6. 2026Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedulescited by 1
  7. 2024You Only Accept Samples Once: Fast, Self-Correcting Stochastic Variational Inferencecited by 0
  8. 2026Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Spacecited by 0
  9. 2026SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batchingcited by 0
  10. 2026Large-scale empirical tuning and comparison of default optimizers for variational inferencecited by 0