# A Stein variational Newton method

arXiv: https://arxiv.org/abs/1806.03085 · HTML: https://mlsanity.com/paper/1806.03085/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2018.

## Built on by (most cited first, 30 of 50)

- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [Semi-supervised Invertible Neural Operators for Bayesian Inverse Problems](https://mlsanity.com/paper/2209.02772/) (2022)
- [Projected Wasserstein gradient descent for high-dimensional Bayesian inference](https://mlsanity.com/paper/2102.06350/) (2021)
- [SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity](https://mlsanity.com/paper/2401.00604/) (2023)
- [On Stein Variational Neural Network Ensembles](https://mlsanity.com/paper/2106.10760/) (2021)
- [Outlier-robust Kalman Filtering through Generalised Bayes](https://mlsanity.com/paper/2405.05646/) (2024)
- [Particle-based Variational Inference with Preconditioned Functional Gradient Flow](https://mlsanity.com/paper/2211.13954/) (2022)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Neural Variational Gradient Descent](https://mlsanity.com/paper/2107.10731/) (2021)
- [Sampling with Mirrored Stein Operators](https://mlsanity.com/paper/2106.12506/) (2021)
- [Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition](https://mlsanity.com/paper/2206.00508/) (2022)
- [Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance](https://mlsanity.com/paper/2302.11024/) (2023)
- [Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent](https://mlsanity.com/paper/2305.14076/) (2023)
- [Grassmann Stein Variational Gradient Descent](https://mlsanity.com/paper/2202.03297/) (2022)
- [Tuning-Free Maximum Likelihood Training of Latent Variable Models via Coin Betting](https://mlsanity.com/paper/2305.14916/) (2023)
- [Adaptive operator learning for infinite-dimensional Bayesian inverse problems](https://mlsanity.com/paper/2310.17844/) (2023)
- [Stein transport for Bayesian inference](https://mlsanity.com/paper/2409.01464/) (2024)
- [Annealing Flow Generative Models Towards Sampling High-Dimensional and Multi-Modal Distributions](https://mlsanity.com/paper/2409.20547/) (2024)
- [Particle-based Variational Inference with Generalized Wasserstein Gradient Flow](https://mlsanity.com/paper/2310.16516/) (2023)
- [Optimal Neural Network Approximation of Wasserstein Gradient Direction via Convex Optimization](https://mlsanity.com/paper/2205.13098/) (2022)
- [Batch Bayesian optimisation via density-ratio estimation with guarantees](https://mlsanity.com/paper/2209.10715/) (2022)
- [Coin Sampling: Gradient-Based Bayesian Inference without Learning Rates](https://mlsanity.com/paper/2301.11294/) (2023)
- [Kernel Semi-Implicit Variational Inference](https://mlsanity.com/paper/2405.18997/) (2024)
- [Performance analysis of greedy algorithms for minimising a Maximum Mean Discrepancy](https://mlsanity.com/paper/2101.07564/) (2021)
- [Multilevel Stein variational gradient descent with applications to Bayesian inverse problems](https://mlsanity.com/paper/2104.01945/) (2021)
- [Stein Variational Gradient Descent with Multiple Kernel](https://mlsanity.com/paper/2107.09338/) (2021)
- [A stochastic Stein Variational Newton method](https://mlsanity.com/paper/2204.09039/) (2022)
- [A Note on the Convergence of Mirrored Stein Variational Gradient Descent under (L0,L1)-Smoothness Condition](https://mlsanity.com/paper/2206.09709/) (2022)
- [Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning](https://mlsanity.com/paper/2405.09536/) (2024)

## Abstract (arXiv)

Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reproducing kernel Hilbert space. In this paper, we accelerate and generalize the SVGD algorithm by including second-order information, thereby approximating a Newton-like iteration in function space. We also show how second-order information can lead to more effective choices of kernel. We observe significant computational gains over the original SVGD algorithm in multiple test cases.