# Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control

arXiv: https://arxiv.org/abs/2608.07433 · HTML: https://mlsanity.com/paper/2608.07433/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 17 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems](https://mlsanity.com/paper/1812.08305/) (2018)
- [Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon](https://mlsanity.com/paper/2011.10300/) (2020)
- [Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem](https://mlsanity.com/paper/1912.11899/) (2019)
- [Policy Optimization as Wasserstein Gradient Flows](https://mlsanity.com/paper/1808.03030/) (2018)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Efficient Wasserstein Natural Gradients for Reinforcement Learning](https://mlsanity.com/paper/2010.05380/) (2020)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Wasserstein Policy Optimization](https://mlsanity.com/paper/2505.00663/) (2025)
- [Kernelized Wasserstein Natural Gradient](https://mlsanity.com/paper/1910.09652/) (2019)
- [Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actions](https://mlsanity.com/paper/2210.11137/) (2022)
- [Provably Convergent Policy Optimization via Metric-aware Trust Region Methods](https://mlsanity.com/paper/2306.14133/) (2023)
- [Wasserstein Proximal Policy Gradient](https://mlsanity.com/paper/2603.02576/) (2026)
- [A note on convergence of Wasserstein policy optimization](https://mlsanity.com/paper/2605.22622/) (2026)
- [Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning](https://mlsanity.com/paper/2605.26078/) (2026)

## Abstract (arXiv)

Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space. We study entropy-regularized discounted linear-quadratic (LQ) control. A Bellman verification argument shows that the unrestricted problem has a linear-Gaussian optimal policy, and the discounted-occupancy-weighted statewise Wasserstein gradient is tangent to this policy class. WPG therefore reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance. We prove that this ODE is globally well posed and converges exponentially from every admissible initialization. For each fixed LQ problem, the exponent has a positive limit as the entropy temperature tends to zero and contains no perturbative factor of the form exp(-c/τ), while retaining the usual dependence on the conditioning of the control problem.