mlsanityData to 7 Oct 2026

SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching

arXiv 2605.07531 ↗First posted 8 May 2026cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Black-Box Variational Inference (BBVI) typically relies on Stochastic Gradient Descent (SGD) to optimize the Evidence Lower Bound (ELBO). However, the stochastic gradients in BBVI inherently exhibit unbounded variance, violating standard assumptions and instead satisfying the weaker Blum-Gladyshev (BG) condition, where variance grows quadratically with distance from the optimum. In this paper, we bridge the gap between stochastic optimization theory and the practical instances of BBVI. Focusing on the broad elliptic location-scale family of parameterized distributions, we offer two main contributions. First, we prove the existence of an ELBO solution, a foundational property usually assumed a priori in the literature. Second, we establish comprehensive convergence guarantees spanning finite-time and asymptotic regimes for Minibatch Projected SGD (PSGD) equipped with dynamic batching and preconditioning under the BG condition. Our theoretical framework demonstrates that dynamic batching combined with preconditioning systematically enables rigorous guarantees even in complex settings. We illustrate our theoretical findings with numerical results, highlighting the efficacy of our approach for modern inference tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

25 of 46 references.

References · 25

  1. 2016Variational Inference: A Review for Statisticianscited by 1,466
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  4. 2013Black Box Variational Inferencecited by 317
  5. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  6. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  7. 2015Black box variational inference for state space modelscited by 31
  8. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  9. 2023Provable convergence guarantees for black-box variational inferencecited by 29
  10. 2023Unconstrained Online Learning with Unbounded Lossescited by 25
  11. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  12. 2024Batch and match: black-box variational inference with a score-based divergencecited by 22
13 more
  1. 2019Provable Gradient Variance Guarantees for Black-Box Variational Inferencecited by 21
  2. 2023Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Boxcited by 17
  3. 2023Variational Inference with Gaussian Score Matchingcited by 17
  4. 2024Dealing with unbounded gradients in stochastic saddle-point optimizationcited by 15
  5. 2018Boosting Black Box Variational Inferencecited by 14
  6. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 13
  7. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8
  8. 2024Batch, match, and patch: low-rank approximations for score-based variational inferencecited by 8
  9. 2014Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Netscited by 5
  10. 2023U-Statistics for Importance-Weighted Variational Inferencecited by 3
  11. 2025Natural Gradient VI: Guarantees for Non-Conjugate Modelscited by 3
  12. 2026Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variancecited by 2
  13. 2026Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedulescited by 1

Built on by

Not cited yet.