mlsanityData to 14 Oct 2026

Flipping-based Policy for Chance-Constrained Markov Decision Processes

arXiv 2410.06474 ↗First posted 9 Oct 2024cs.LG math.OCcited by 3

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safe reinforcement learning (RL) is a promising approach for many real-world decision-making problems where ensuring safety is a critical necessity. In safe RL research, while expected cumulative safety constraints (ECSCs) are typically the first choices, chance constraints are often more pragmatic for incorporating safety under uncertainties. This paper proposes a textitflipping-based policy for Chance-Constrained Markov Decision Processes (CCMDPs). The flipping-based policy selects the next action by tossing a potentially distorted coin between two action candidates. The probability of the flip and the two action candidates vary depending on the state. We establish a Bellman equation for CCMDPs and further prove the existence of a flipping-based policy within the optimal solution sets. Since solving the problem with joint chance constraints is challenging in practice, we then prove that joint chance constraints can be approximated into Expected Cumulative Safety Constraints (ECSCs) and that there exists a flipping-based policy in the optimal solution sets for constrained MDPs with ECSCs. As a specific instance of practical implementations, we present a framework for adapting constrained policy optimization to train a flipping-based policy. This framework can be applied to other safe RL algorithms. We demonstrate that the flipping-based policy can improve the performance of the existing safe RL algorithms under the same limits of safety constraints on Safety Gym benchmarks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 52 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

31 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation2 May
  2. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning3 Feb

Other references · 29

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2017Safe Reinforcement Learning via Shieldingcited by 449
  4. 2020Projection-Based Constrained Policy Optimizationcited by 203
  5. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  6. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  7. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  8. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  9. 2019Data-driven Economic NMPC using Reinforcement Learningcited by 68
  10. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  11. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  12. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
17 more
  1. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  2. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 35
  3. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 34
  4. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 31
  5. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 28
  6. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  7. 2021Safe Reinforcement Learning with Chance-constrained Model Predictive Controlcited by 20
  8. 2023A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraintscited by 17
  9. 2021Safe Chance Constrained Reinforcement Learning for Batch Process Controlcited by 11
  10. 2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 11
  11. 2022Provable Safe Reinforcement Learning with Binary Feedbackcited by 7
  12. 2023Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Explorationcited by 7
  13. 2019Computing Probabilistic Controlled Invariant Setscited by 6
  14. 2022Data-Driven Chance Constrained Control using Kernel Distribution Embeddingscited by 5
  15. 2013Reinforcement learning with restrictions on the action setcited by 1
  16. 2022Nonconvex and Nonsmooth Approaches for Affine Chance-Constrained Stochastic Programscited by 1
  17. 2022Approximate Methods for Solving Chance Constrained Linear Programs in Probability Measure Spacecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategiescited by 4
  2. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  3. 2026Learning Chance-Constrained MDPs with Bellman Distributional Certificatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
  2. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety
  3. 2025Probabilistic Shielding for Safe Reinforcement Learning
  4. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  5. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  6. 2025Ensuring Safety in an Uncertain Environment: Constrained MDPs via Stochastic Thresholds
  7. 2024Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning
  8. 2024Long-term Safe Reinforcement Learning with Binary Feedback