mlsanityData to 14 Oct 2026

Learning Optimal Deterministic Policies with Stochastic Policy Gradients

arXiv 2405.02235 ↗First posted 3 May 2024cs.LGcited by 12

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. They learn stochastic parametric (hyper)policies by either exploring in the space of actions or in the space of parameters. Stochastic controllers, however, are often undesirable from a practical perspective because of their lack of robustness, safety, and traceability. In common practice, stochastic (hyper)policies are learned only to deploy their deterministic version. In this paper, we make a step towards the theoretical understanding of this practice. After introducing a novel framework for modeling this scenario, we study the global convergence to the best deterministic policy, under (weak) gradient domination assumptions. Then, we illustrate how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy. Finally, we quantitatively compare action-based and parameter-based exploration, giving a formal guise to intuitive results.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 56 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

24 of 56 references.

References · 24

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  3. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  4. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  6. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  7. 2018Understanding the impact of entropy on policy optimizationcited by 176
  8. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  9. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  10. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  11. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  12. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
12 more
  1. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  2. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 59
  3. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  4. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  5. 2018Policy Optimization via Importance Samplingcited by 47
  6. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 40
  7. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  8. 2019Smoothing Policies and Safe Policy Gradientscited by 21
  9. 2022Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Functioncited by 20
  10. 2020Zeroth-order Deterministic Policy Gradientcited by 9
  11. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  12. 2022Truly Deterministic Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 7.

12 papers.

  1. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  2. 2024Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learningcited by 2
  3. 2025Reusing Trajectories in Policy Gradients Enables Fast Convergencecited by 1
  4. 2026How Log-Barrier Helps Exploration in Policy Optimizationcited by 1
  5. 2026Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matchingcited by 1
  6. 2024Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPscited by 0
  7. 2025Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustnesscited by 0
  8. 2026Analyzing Symbolic Properties for DRL Agents in Systems and Networkingcited by 0
  9. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flowscited by 0
  10. 2026SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraintscited by 0
  11. 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0
  12. 2026Variance-Reduced Policy Gradient is Nearly Minimax-Optimalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes
  2. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  3. 2025Equivalence of stochastic and deterministic policy gradients
  4. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  5. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
  6. 2024Matrix Low-Rank Approximation For Policy Gradient Methods
  7. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning
  8. 2026Bayesian policy gradient and actor-critic algorithms