mlsanityData to 14 Oct 2026

A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization

arXiv 2110.08923 ↗First posted 17 Oct 2021cs.LG math.OCcited by 34

Not on the map. Why?

Abstract (arXiv)

We study entropy-regularized constrained Markov decision processes (CMDPs) under the soft-max parameterization, in which an agent aims to maximize the entropy-regularized value function while satisfying constraints on the expected total utility. By leveraging the entropy regularization, our theoretical analysis shows that its Lagrangian dual function is smooth and the Lagrangian duality gap can be decomposed into the primal optimality gap and the constraint violation. Furthermore, we propose an accelerated dual-descent method for entropy-regularized CMDPs. We prove that our method achieves the global convergence rate widetildeO(1/T) for both the optimality gap and the constraint violation for entropy-regularized CMDPs. A discussion about a linear convergence rate for CMDPs with a single constraint is also provided.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 16 of its 39 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

16 of 39 references.

References · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  6. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  7. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  8. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  9. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  10. 2020Exploration-Exploitation in Constrained MDPscited by 149
  11. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  12. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
4 more
  1. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  2. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  3. 2015Complexity of first order inexact Lagrangian and penalty methods for conic convex programmingcited by 7
  4. 2019Non-Cooperative Inverse Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 5; 2024: 12; 2025: 5; 2026: 4.

34 papers.

Sort:

By area:

  1. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
  2. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 27
  3. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  4. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 23
  5. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  6. 2022Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Gamescited by 18
  7. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  8. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15
  9. 2022Robust Constrained Reinforcement Learningcited by 15
  10. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  11. 2023Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learningcited by 14
  12. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
22 more
  1. 2022Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraintscited by 10
  2. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  3. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 8
  4. 2023Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processescited by 7
  5. 2024A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guaranteescited by 6
  6. 2025Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximationcited by 6
  7. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  8. 2022Algorithm for Constrained Markov Decision Process with Linear Convergencecited by 4
  9. 2024Finite-Time Error Analysis of Soft Q-Learning: Switching System Approachcited by 3
  10. 2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
  11. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithmcited by 2
  12. 2024Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learningcited by 2
  13. 2024The Feasibility Theory of Constrained Reinforcement Learning: A Tutorial Studycited by 1
  14. 2024Last-Iterate Convergence of General Parameterized Policies in Constrained MDPscited by 1
  15. 2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
  16. 2026Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Marginscited by 1
  17. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1
  18. 2024Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learningcited by 0
  19. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  20. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  21. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0
  22. 2026Fast Regularized Policy Mirror Descent with One-Step TD Updatescited by 0