ELBOing Stein: Variational Bayes with Stein Mixture Inference
Not on the map. Why?
Abstract (arXiv)
Stein variational gradient descent (SVGD) [Liu and Wang, 2016] performs approximate Bayesian inference by representing the posterior with a set of particles. However, SVGD suffers from variance collapse, i.e. poor predictions due to underestimating uncertainty [Ba et al., 2021], even for moderately-dimensional models such as small Bayesian neural networks (BNNs). To address this issue, we generalize SVGD by letting each particle parameterize a component distribution in a mixture model. Our method, Stein Mixture Inference (SMI), optimizes a lower bound to the evidence (ELBO) and introduces user-specified guides parameterized by particles. SMI extends the Nonlinear SVGD framework [Wang and Liu, 2019] to the case of variational Bayes. SMI effectively avoids variance collapse, judging by a previously described test developed for this purpose, and performs well on standard data sets. In addition, SMI requires considerably fewer particles than SVGD to accurately estimate uncertainty for small BNNs. The synergistic combination of NSVGD, ELBO optimization and user-specified guides establishes a promising approach towards variational Bayesian inference in the case of tall and wide data.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 30 Oct 2024.
Builds on
38 of 57 references.
References · 38
- 2013Intriguing properties of neural networkscited by 5,114
- 2016Variational Inference: A Review for Statisticianscited by 1,466
- 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
- 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
- 2013Black Box Variational Inferencecited by 317
- 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
- 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
- 2016Automatic Differentiation Variational Inferencecited by 159
- 2017Stein Variational Gradient Descent as Gradient Flowcited by 149
- 2017Measuring Sample Quality with Kernelscited by 119
- 2015Practical Bayesian model evaluation using leave-one-out cross-validation and WAICcited by 110
- 2021Repulsive Deep Ensembles are Bayesiancited by 103
26 more
- 2019Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMCcited by 90
- 2015Hierarchical Variational Modelscited by 89
- 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
- 2019On the geometry of Stein variational gradient descentcited by 82
- 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
- 2020SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergencecited by 64
- 2018A Stein variational Newton methodcited by 49
- 2017Message Passing Stein Variational Gradient Descentcited by 47
- 2017Optimal transport for Gaussian mixture modelscited by 45
- 2018A Unified Particle-Optimization Framework for Scalable Bayesian Samplingcited by 44
- 2016Variational Boosting: Iteratively Refining Posterior Approximationscited by 36
- 2018Scaling limit of the Stein variational gradient descent: the mean field regimecited by 35
- 2019Stein Variational Gradient Descent With Matrix-Valued Kernelscited by 32
- 2017Learning to Draw Samples with Amortized Stein Variational Gradient Descentcited by 26
- 2017Riemannian Stein Variational Gradient Descent for Bayesian Inferencecited by 24
- 2021Annealed Stein Variational Gradient Descentcited by 24
- 2022A Finite-Particle Convergence Rate for Stein Variational Gradient Descentcited by 22
- 2022Particle-based Variational Inference with Preconditioned Functional Gradient Flowcited by 20
- 2018Stochastic Gradient MCMC with Repulsive Forcescited by 18
- 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
- 2018Stein Variational Gradient Descent as Moment Matchingcited by 15
- 2014Variational Particle Approximationscited by 13
- 2018On the Importance of Strong Baselines in Bayesian Deep Learningcited by 12
- 2017Stein Variational Adaptive Importance Samplingcited by 7
- 2017A Divergence Bound for Hybrids of MCMC and Variational Inference and an Application to Langevin Dynamics and SGVIcited by 4
- 2019Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variablescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.