mlsanityData to 14 Oct 2026

Equivalence of stochastic and deterministic policy gradients

arXiv 2505.23244 ↗First posted 29 May 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy gradients in continuous control have been derived for both stochastic and deterministic policies. Here we study the relationship between the two. In a widely-used family of MDPs involving Gaussian control noise and quadratic control costs, we show that the stochastic and deterministic policy gradients, natural gradients, and state value functions are identical; while the state-control value functions are different. We then develop a general procedure for constructing an MDP with deterministic policy that is equivalent to a given MDP with stochastic policy. The controls of this new MDP are the sufficient statistics of the stochastic policy in the original MDP. Our results suggest that policy gradient methods can be unified by approximating state value functions rather than state-control value functions.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 4 of its 13 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

4 of 13 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Wasserstein Policy Optimization1 May

Other references · 3

  1. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  2. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  3. 2019On Policy Gradientscited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Dynamical System Optimizationcited by 1
  2. 2025Distributions as Actions: A Unified Framework for Diverse Action Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Truly Deterministic Policy Optimization
  2. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradients
  3. 2024Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs
  4. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  5. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning
  6. 2024Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action
  7. 2026Soft Deterministic Policy Gradient with Gaussian Smoothing
  8. 2026(Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator