mlsanityData to 7 Oct 2026

Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs

arXiv 2408.10015 ↗First posted 19 Aug 2024cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the problem of computing deterministic optimal policies for constrained Markov decision processes (MDPs) with continuous state and action spaces, which are widely encountered in constrained dynamical systems. Designing deterministic policy gradient methods in continuous state and action spaces is particularly challenging due to the lack of enumerable state-action pairs and the adoption of deterministic policies, hindering the application of existing policy gradient methods. To this end, we develop a deterministic policy gradient primal-dual method to find an optimal deterministic policy with non-asymptotic convergence. Specifically, we leverage regularization of the Lagrangian of the constrained MDP to propose a deterministic policy gradient primal-dual (D-PGPD) algorithm that updates the deterministic policy via a quadratic-regularized gradient ascent step and the dual variable via a quadratic-regularized gradient descent step. We prove that the primal-dual iterates of D-PGPD converge at a sub-linear rate to an optimal regularized primal-dual pair. We instantiate D-PGPD with function approximation and prove that the primal-dual iterates of D-PGPD converge at a sub-linear rate to an optimal regularized primal-dual pair, up to a function approximation error. Furthermore, we demonstrate the effectiveness of our method in two continuous control problems: robot navigation and fluid control. This appears to be the first work that proposes a deterministic policy search method for continuous-space constrained MDPs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

27 of 49 references.

References · 27

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Reward Constrained Policy Optimizationcited by 397
  3. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  4. 2020Exploration-Exploitation in Constrained MDPscited by 146
  5. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  6. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  7. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  8. 2021Reward is enough for convex MDPscited by 82
  9. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  10. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
  11. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  12. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
15 more
  1. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  2. 2022Policy Optimization over General State and Action Spacescited by 21
  3. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  4. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradientscited by 10
  5. 2020Zeroth-order Deterministic Policy Gradientcited by 9
  6. 2020Alternating Direction Method of Multipliers for Constrained Iterative LQR in Autonomous Drivingcited by 9
  7. 2018Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert Spacescited by 7
  8. 2020Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulatorcited by 7
  9. 2024Deterministic Policies for Constrained Reinforcement Learning in Polynomial Timecited by 5
  10. 2020Risk-Constrained Linear-Quadratic Regulatorscited by 4
  11. 2021Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Costcited by 3
  12. 2019Prescribed-time convergence with input constraints: A control Lyapunov function based approachcited by 2
  13. 2019The Quadratic-Quadratic Regulator Problem: Approximating feedback controls for quadratic-in-state nonlinear systemscited by 2
  14. 2015Solving the Infinite-horizon Constrained LQR Problem using Accelerated Dual Proximal Methodscited by 1
  15. 2018Sufficiency of Deterministic Policies for Atomless Discounted and Uniformly Absorbing MDPs with Multiple Criteriacited by 1

Built on by

Not cited yet.