# Stein Variational Gradient Descent with Multiple Kernel

arXiv: https://arxiv.org/abs/2107.09338 · HTML: https://mlsanity.com/paper/2107.09338/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 20 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation](https://mlsanity.com/paper/1602.03253/) (2016)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Measuring Sample Quality with Kernels](https://mlsanity.com/paper/1703.01717/) (2017)
- [Measuring Sample Quality with Stein's Method](https://mlsanity.com/paper/1506.03039/) (2015)
- [A Non-Asymptotic Analysis for Stein Variational Gradient Descent](https://mlsanity.com/paper/2006.09797/) (2020)
- [Gradient Estimators for Implicit Models](https://mlsanity.com/paper/1705.07107/) (2017)
- [A Stein variational Newton method](https://mlsanity.com/paper/1806.03085/) (2018)
- [A Unified Particle-Optimization Framework for Scalable Bayesian Sampling](https://mlsanity.com/paper/1805.11659/) (2018)
- [Projected Stein Variational Gradient Descent](https://mlsanity.com/paper/2002.03469/) (2020)
- [Stein Variational Gradient Descent With Matrix-Valued Kernels](https://mlsanity.com/paper/1910.12794/) (2019)
- [Learning to Draw Samples with Amortized Stein Variational Gradient Descent](https://mlsanity.com/paper/1707.06626/) (2017)
- [Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory](https://mlsanity.com/paper/1809.01293/) (2018)
- [Stein Variational Inference for Discrete Distributions](https://mlsanity.com/paper/2003.00605/) (2020)
- [Stein Variational Gradient Descent Without Gradient](https://mlsanity.com/paper/1806.02775/) (2018)
- [Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent](https://mlsanity.com/paper/2106.07832/) (2021)
- [Understanding and Accelerating Particle-Based Variational Inference](https://mlsanity.com/paper/1807.01750/) (2018)
- [Self-weighted Multiple Kernel Learning for Graph-based Clustering and Semi-supervised Classification](https://mlsanity.com/paper/1806.07697/) (2018)

## Built on by (most cited first, 4 of 4)

- [Improving Pareto Set Learning for Expensive Multi-objective Optimization via Stein Variational Hypernetworks](https://mlsanity.com/paper/2412.17312/) (2024)
- [A Framework for Controllable Multi-objective Learning with Annealed Stein Variational Hypernetworks](https://mlsanity.com/paper/2506.06715/) (2025)
- [Adaptive Kernel Selection for Stein Variational Gradient Descent](https://mlsanity.com/paper/2510.02067/) (2025)
- [Nonlocal Hamiltonian Dynamics on Sparse L'evy Graphs: Spectral Analysis and Multimodal Sampling](https://mlsanity.com/paper/2610.06904/) (2026)

## Abstract (arXiv)

Stein variational gradient descent (SVGD) and its variants have shown promising successes in approximate inference for complex distributions. In practice, we notice that the kernel used in SVGD-based methods has a decisive effect on the empirical performance. Radial basis function (RBF) kernel with median heuristics is a common choice in previous approaches, but unfortunately this has proven to be sub-optimal. Inspired by the paradigm of Multiple Kernel Learning (MKL), our solution to this flaw is using a combination of multiple kernels to approximate the optimal kernel, rather than a single one which may limit the performance and flexibility. Specifically, we first extend Kernelized Stein Discrepancy (KSD) to its multiple kernels view called Multiple Kernelized Stein Discrepancy (MKSD) and then leverage MKSD to construct a general algorithm Multiple Kernel SVGD (MK-SVGD). Further, MKSVGD can automatically assign a weight to each kernel without any other parameters, which means that our method not only gets rid of optimal kernel dependence but also maintains computational efficiency. Experiments on various tasks and models demonstrate that our proposed method consistently matches or outperforms the competing methods.