mlsanityData to 7 Oct 2026

Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference

arXiv 2303.10472 ↗First posted 18 Mar 2023cs.LG math.OC stat.COcited by 6

Not on the map. Why?

Abstract (arXiv)

Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the ABC condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 18 Mar 2023.

Builds on

42 of 61 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Variational Inference: A Review for Statisticianscited by 1,466
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  4. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  6. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  7. 2013Black Box Variational Inferencecited by 317
  8. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  9. 2017Advances in Variational Inferencecited by 236
  10. 2019SGD: General Analysis and Improved Ratescited by 206
  11. 2016Automatic Differentiation Variational Inferencecited by 159
  12. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
30 more
  1. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  2. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  3. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  4. 2017TensorFlow Distributionscited by 105
  5. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
  6. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  7. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 66
  8. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  9. 2018Yes, but Did It Work?: Evaluating Variational Inferencecited by 52
  10. 2018Variance reduction properties of the reparameterization trickcited by 41
  11. 2019Near-Optimal Methods for Minimizing Star-Convex Functions and Beyondcited by 36
  12. 2021Challenges and Opportunities in High-dimensional Variational Inferencecited by 35
  13. 2018Stochastic Quasi-Gradient Methods: Variance Reduction via Jacobian Sketchingcited by 34
  14. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 32
  15. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  16. 2017Reducing Reparameterization Gradient Variancecited by 27
  17. 2018Quasi-Monte Carlo Variational Inferencecited by 24
  18. 2018Using Large Ensembles of Control Variates for Variational Inferencecited by 21
  19. 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
  20. 2020Robust, Accurate Stochastic Optimization for Variational Inferencecited by 21
  21. 2020Non-exponentially weighted aggregation: regret bounds for unbounded loss functionscited by 18
  22. 2019A Generalization Bound for Online Variational Inferencecited by 17
  23. 2019Multilevel Monte Carlo Variational Inferencecited by 14
  24. 2022A Unified Convergence Theorem for Stochastic Optimization Methodscited by 12
  25. 2022A Framework for Improving the Reliability of Black-box Variational Inferencecited by 9
  26. 2015Fast Second-Order Stochastic Backpropagation for Variational Inferencecited by 8
  27. 2020On The Convergence of First Order Methods for Quasar-Convex Optimizationcited by 8
  28. 2022Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selectioncited by 7
  29. 2019A Rule for Gradient Estimator Selection, with an Application to Variational Inferencecited by 5
  30. 2023Parametrization Cookbook: A set of Bijective Parametrizations for using Machine Learning methods in Statistical Inferencecited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 1.

6 papers.

  1. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  2. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  3. 2024Provably Scalable Black-Box Variational Inference with Structured Variational Familiescited by 4
  4. 2024Stochastic variance-reduced Gaussian variational inference on the Bures-Wasserstein manifoldcited by 2
  5. 2024Understanding and mitigating difficulties in posterior predictive evaluationcited by 1
  6. 2025Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inferencecited by 1