Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference
Not on the map. Why?
Abstract (arXiv)
We prove that, given a mean-field location-scale variational family, black-box variational inference (BBVI) with the reparametrization gradient converges at a rate that is nearly independent of explicit dimension dependence. Specifically, for a d-dimensional strongly log-concave and log-smooth target, the number of iterations for BBVI with a sub-Gaussian family to obtain a solution ε-close to the global optimum has a dimension dependence of O(log d). This is a significant improvement over the O(d) dependence of full-rank location-scale families. For heavy-tailed families, we prove a weaker O(d2/k) dependence, where k is the number of finite moments of the family. Additionally, if the Hessian of the target log-density is constant, the complexity is free of any explicit dimension dependence. We also prove that our bound on the gradient variance, which is key to our result, cannot be improved using only spectral bounds on the Hessian of the target log-density.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 61 of its 102 references that are in the map. First posted on arXiv 27 May 2025.
Builds on
61 of 102 references.
References · 61
- 2016Variational Inference: A Review for Statisticianscited by 1,466
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
- 2013Black Box Variational Inferencecited by 317
- 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
- 2019SGD: General Analysis and Improved Ratescited by 206
- 2016Automatic Differentiation Variational Inferencecited by 159
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 158
- 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
- 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
- 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
- 2022Variational inference via Wasserstein gradient flowscited by 84
38 more
- 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
- 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 68
- 2017Concentration of tempered posteriors and of their variational approximationscited by 66
- 2017Frequentist Consistency of Variational Bayescited by 60
- 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
- 2017Convergence Rates of Variational Posterior Distributionscited by 45
- 2018Variance reduction properties of the reparameterization trickcited by 41
- 2017Covariances, Robustness, and Variational Bayescited by 39
- 2020Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimizationcited by 37
- 2020Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimizationcited by 36
- 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
- 2013Automated Variational Inference in Probabilistic Programmingcited by 31
- 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
- 2023Provable convergence guarantees for black-box variational inferencecited by 29
- 2017Reducing Reparameterization Gradient Variancecited by 27
- 2021Pathfinder: Parallel quasi-Newton variational inferencecited by 26
- 2018Quasi-Monte Carlo Variational Inferencecited by 24
- 2017Gaussian variational approximation with a factor covariance structurecited by 22
- 2019Variational Bayes under Model Misspecificationcited by 22
- 2023On the Convergence of Black-Box Variational Inferencecited by 22
- 2018Using Large Ensembles of Control Variates for Variational Inferencecited by 21
- 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
- 2023Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein spacecited by 20
- 2015Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functionscited by 19
- 2017Theoretical and Computational Guarantees of Mean Field Variational Inference for Community Detectioncited by 19
- 2019Fast and Furious Convergence: Stochastic Second Order Methods under Interpolationcited by 18
- 2023On the Convergence of Coordinate Ascent Variational Inferencecited by 18
- 2023Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Boxcited by 17
- 2019Multilevel Monte Carlo Variational Inferencecited by 14
- 2024Convergence of coordinate ascent variational inference for log-concave measures via optimal transportcited by 13
- 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 13
- 2018An Instability in Variational Inference for Topic Modelscited by 11
- 2020Approximation Based Variance Reduction for Reparameterization Gradientscited by 10
- 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
- 2023The Shrinkage-Delinkage Trade-off: An Analysis of Factorized Gaussian Approximations for Variational Inferencecited by 9
- 2024Convergence rate of random scan Coordinate Ascent Variational Inference under log-concavitycited by 9
- 2024Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrixcited by 9
- 2015Fast Second-Order Stochastic Backpropagation for Variational Inferencecited by 8
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.