mlsanityData to 14 Oct 2026

Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent

arXiv 2305.14076 ↗First posted 23 May 2023math.ST cs.LG math.PRcited by 13

Not on the map. Why?

Abstract (arXiv)

Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. Despite its wide usage, understanding the theoretical properties of SVGD has remained a challenging problem. For sampling from a Gaussian target, the SVGD dynamics with a bilinear kernel will remain Gaussian as long as the initializer is Gaussian. Inspired by this fact, we undertake a detailed theoretical study of the Gaussian-SVGD, i.e., SVGD projected to the family of Gaussian distributions via the bilinear kernel, or equivalently Gaussian variational inference (GVI) with SVGD. We present a complete picture by considering both the mean-field PDE and discrete particle systems. When the target is strongly log-concave, the mean-field Gaussian-SVGD dynamics is proven to converge linearly to the Gaussian distribution closest to the target in KL divergence. In the finite-particle setting, there is both uniform in time convergence to the mean-field limit and linear convergence in time to the equilibrium if the target is Gaussian. In the general case, we propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unified framework. Interestingly, one of the new particle-based instance from this framework empirically outperforms existing approaches. Our results make concrete contributions towards obtaining a deeper understanding of both SVGD and GVI.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 50 of its 90 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

50 of 90 references.

References · 50

  1. 2015Variational Inference with Normalizing Flowscited by 2,170
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  5. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  6. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
  7. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  8. 2017Stein Variational Gradient Descent as Gradient Flowcited by 152
  9. 2016A Kernel Test of Goodness of Fitcited by 143
  10. 2019Maximum Mean Discrepancy Gradient Flowcited by 127
  11. 2017Measuring Sample Quality with Kernelscited by 119
  12. 2022Variational inference via Wasserstein gradient flowscited by 85
38 more
  1. 2019On the geometry of Stein variational gradient descentcited by 82
  2. 2017Concentration of tempered posteriors and of their variational approximationscited by 66
  3. 2020SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergencecited by 64
  4. 2020A Non-Asymptotic Analysis for Stein Variational Gradient Descentcited by 61
  5. 2021Kernel Stein Discrepancy Descentcited by 57
  6. 2021Optimizing Functionals on the Space of Probabilities with Input Convex Neural Networkscited by 53
  7. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  8. 2018A Stein variational Newton methodcited by 50
  9. 2017Message Passing Stein Variational Gradient Descentcited by 47
  10. 2018Convergence and Concentration of Empirical Measures under Wasserstein Distance in Unbounded Functional Spacescited by 41
  11. 2019Accelerated Information Gradient flowcited by 39
  12. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  13. 2018Scaling limit of the Stein variational gradient descent: the mean field regimecited by 35
  14. 2020Projected Stein Variational Gradient Descentcited by 33
  15. 2019Stein Variational Gradient Descent With Matrix-Valued Kernelscited by 32
  16. 2020Stochastic Stein Discrepanciescited by 30
  17. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  18. 2017Learning to Draw Samples with Amortized Stein Variational Gradient Descentcited by 28
  19. 2017Riemannian Stein Variational Gradient Descent for Bayesian Inferencecited by 24
  20. 2020Information Newton's flow: second-order optimization method in probability spacecited by 24
  21. 2021Projected Wasserstein gradient descent for high-dimensional Bayesian inferencecited by 24
  22. 2022A Finite-Particle Convergence Rate for Stein Variational Gradient Descentcited by 22
  23. 2021A Convergence Theory for SVGD in the Population Limit under Talagrand's Inequality T1cited by 21
  24. 2022Mean-field Variational Inference via Wasserstein Gradient Flowcited by 19
  25. 2022Regularized Stein Variational Gradient Flowcited by 19
  26. 2019A Generalization Bound for Online Variational Inferencecited by 17
  27. 2021Sampling with Mirrored Stein Operatorscited by 17
  28. 2022Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Conditioncited by 16
  29. 2023Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariancecited by 16
  30. 2016Gaussian variational approximation with sparse precision matricescited by 15
  31. 2018Stein Variational Gradient Descent as Moment Matchingcited by 15
  32. 2019Accelerated Flow for Probability Distributionscited by 15
  33. 2020Variational Inference with Continuously-Indexed Normalizing Flowscited by 15
  34. 2022Grassmann Stein Variational Gradient Descentcited by 12
  35. 2018Gaussian variational approximation for high-dimensional state space modelscited by 10
  36. 2022When Optimal Transport Meets Information Geometrycited by 10
  37. 2021Approximating Bayes in the 21st Centurycited by 6
  38. 2022Stochastic Gradient Descent for Barycenters in Wasserstein Spacecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 4; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023Sampling via Gradient Flows in the Space of Probability Measurescited by 20
  2. 2024Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descentcited by 14
  3. 2023Provably Fast Finite Particle Variants of SVGD via Virtual Particle Stochastic Approximationcited by 6
  4. 2025Accelerated Stein Variational Gradient Flowcited by 6
  5. 2026Finite-Particle Rates for Regularized Stein Variational Gradient Descentcited by 5
  6. 2024Bayesian Multi-Task Transfer Learning for Soft Prompt Tuningcited by 3
  7. 2024Long-time asymptotics of noisy SVGD outside the population limitcited by 3
  8. 2024A Particle Algorithm for Mean-Field Variational Inferencecited by 3
  9. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  10. 2025Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributionscited by 2
  11. 2026To discretize continually: Mean shift interacting particle systems for Bayesian inferencecited by 2
  12. 2025Branching Stein Variational Gradient Descent for sampling multimodal distributionscited by 0
1 more
  1. 2026Well-Posed KL-Regularized Control via Wasserstein and Kalman-Wasserstein KL Divergencescited by 0