mlsanityData to 14 Oct 2026

Provably Scalable Black-Box Variational Inference with Structured Variational Families

arXiv 2401.10989 ↗First posted 19 Jan 2024stat.ML cs.LG stat.COcited by 4

Not on the map. Why?

Abstract (arXiv)

Variational families with full-rank covariance approximations are known not to work well in black-box variational inference (BBVI), both empirically and theoretically. In fact, recent computational complexity results for BBVI have established that full-rank variational families scale poorly with the dimensionality of the problem compared to e.g. mean-field families. This is particularly critical to hierarchical Bayesian models with local variables; their dimensionality increases with the size of the datasets. Consequently, one gets an iteration complexity with an explicit O(N2) dependence on the dataset size N. In this paper, we explore a theoretical middle ground between mean-field variational families and full-rank families: structured variational families. We rigorously prove that certain scale matrix structures can achieve a better iteration complexity of Oleft(Nright), implying better scaling with respect to N. We empirically verify our theoretical results on large-scale hierarchical models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 50 of its 84 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

50 of 84 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2013Black Box Variational Inferencecited by 319
  4. 2016Ladder Variational Autoencoderscited by 318
  5. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  6. 2017Advances in Variational Inferencecited by 236
  7. 2019SGD: General Analysis and Improved Ratescited by 207
  8. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  9. 2016Automatic Differentiation Variational Inferencecited by 159
  10. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  11. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 143
  12. 2016Composing graphical models with neural networks for structured representations and fast inferencecited by 113
38 more
  1. 2017Variational Sequential Monte Carlocited by 95
  2. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  3. 2015Hierarchical Variational Modelscited by 89
  4. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  5. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 82
  6. 2016Auxiliary Deep Generative Modelscited by 72
  7. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  8. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  9. 2020Bayesian Workflowcited by 49
  10. 2018Don't Unroll Adjoint: Differentiating SSA-Form Programscited by 45
  11. 2018Variance reduction properties of the reparameterization trickcited by 41
  12. 2017Covariances, Robustness, and Variational Bayescited by 40
  13. 2020Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimizationcited by 37
  14. 2016Linear dynamical neural population models through nonlinear embeddingscited by 36
  15. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  16. 2018Stochastic Quasi-Gradient Methods: Variance Reduction via Jacobian Sketchingcited by 34
  17. 2015Black box variational inference for state space modelscited by 31
  18. 2017Faithful Inversion of Generative Models for Effective Amortized Inferencecited by 29
  19. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  20. 2020Variational Item Response Theory: Fast, Accurate, and Expressivecited by 29
  21. 2023Provable convergence guarantees for black-box variational inferencecited by 29
  22. 2021Pathfinder: Parallel quasi-Newton variational inferencecited by 26
  23. 2017Gaussian variational approximation with a factor covariance structurecited by 22
  24. 2020Automatic structured variational inferencecited by 22
  25. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  26. 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
  27. 2021On the Convergence of Step Decay Step-Size for Stochastic Optimizationcited by 21
  28. 2015Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functionscited by 19
  29. 2021Amortized Variational Inference for Simple Hierarchical Modelscited by 19
  30. 2017Effective Extensible Programming: Unleashing Julia on GPUscited by 16
  31. 2016Gaussian variational approximation with sparse precision matricescited by 15
  32. 2018Gaussian variational approximation for high-dimensional state space modelscited by 10
  33. 2021Automatic variational inference with cascading flowscited by 9
  34. 2015A General Method for Robust Bayesian Modelingcited by 7
  35. 2022Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selectioncited by 7
  36. 2017Proximity Variational Inferencecited by 6
  37. 2023Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inferencecited by 6
  38. 2022Structured Recognition for Generative Models with Explaining Awaycited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Batch, match, and patch: low-rank approximations for score-based variational inferencecited by 8
  2. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  3. 2025Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inferencecited by 1
  4. 2025A Hierarchical Variational Graph Fused Lasso for Recovering Relative Rates in Spatial Compositional Datacited by 0