# Stein Variational Policy Gradient

arXiv: https://arxiv.org/abs/1704.02399 · HTML: https://mlsanity.com/paper/1704.02399/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2017.

## Built on by (most cited first, 30 of 44)

- [Recent Advances in Optimal Transport for Machine Learning](https://mlsanity.com/paper/2306.16156/) (2023)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/2110.15335/) (2021)
- [Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization](https://mlsanity.com/paper/2106.04335/) (2021)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [A Finite-Particle Convergence Rate for Stein Variational Gradient Descent](https://mlsanity.com/paper/2211.09721/) (2022)
- [A Convergence Theory for SVGD in the Population Limit under Talagrand's Inequality T1](https://mlsanity.com/paper/2106.03076/) (2021)
- [Regularized Stein Variational Gradient Flow](https://mlsanity.com/paper/2211.07861/) (2022)
- [S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic](https://mlsanity.com/paper/2405.00987/) (2024)
- [Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition](https://mlsanity.com/paper/2206.00508/) (2022)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [DR2L: Surfacing Corner Cases to Robustify Autonomous Driving via Domain Randomization Reinforcement Learning](https://mlsanity.com/paper/2107.11762/) (2021)
- [CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectories](https://mlsanity.com/paper/2403.13208/) (2024)
- [Reinforcement Learning Algorithms: An Overview and Classification](https://mlsanity.com/paper/2209.14940/) (2022)
- [GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework](https://mlsanity.com/paper/2312.16429/) (2023)
- [ASPiRe:Adaptive Skill Priors for Reinforcement Learning](https://mlsanity.com/paper/2209.15205/) (2022)
- [Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/2206.06719/) (2022)
- [Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2208.12328/) (2022)
- [Provably Fast Finite Particle Variants of SVGD via Virtual Particle Stochastic Approximation](https://mlsanity.com/paper/2305.17558/) (2023)
- [Online reinforcement learning with sparse rewards through an active inference capsule](https://mlsanity.com/paper/2106.02390/) (2021)
- [Loss function based second-order Jensen inequality and its application to particle variational inference](https://mlsanity.com/paper/2106.05010/) (2021)
- [DROMO: Distributionally Robust Offline Model-based Policy Optimization](https://mlsanity.com/paper/2109.07275/) (2021)
- [Learning Rate Free Sampling in Constrained Domains](https://mlsanity.com/paper/2305.14943/) (2023)
- [Long-time asymptotics of noisy SVGD outside the population limit](https://mlsanity.com/paper/2406.11929/) (2024)
- [How to Choose a Reinforcement-Learning Algorithm](https://mlsanity.com/paper/2407.20917/) (2024)
- [Improved Stein Variational Gradient Descent with Importance Weights](https://mlsanity.com/paper/2210.00462/) (2022)
- [Stein Variational Evolution Strategies](https://mlsanity.com/paper/2410.10390/) (2024)

## Abstract (arXiv)

Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this work, we introduce a maximum entropy policy optimization framework which explicitly encourages parameter exploration, and show that this framework can be reduced to a Bayesian inference problem. We then propose a novel Stein variational policy gradient method (SVPG) which combines existing policy gradient methods and a repulsive functional to generate a set of diverse but well-behaved policies. SVPG is robust to initialization and can easily be implemented in a parallel manner. On continuous control problems, we find that implementing SVPG on top of REINFORCE and advantage actor-critic algorithms improves both average return and data efficiency.