# Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertainty

arXiv: https://arxiv.org/abs/2010.04870 · HTML: https://mlsanity.com/paper/2010.04870/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Policy Gradient Method For Robust Reinforcement Learning](https://mlsanity.com/paper/2205.07344/) (2022)
- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [Policy Gradient in Robust MDPs with Global Convergence Guarantee](https://mlsanity.com/paper/2212.10439/) (2022)
- [Robust Average-Reward Markov Decision Processes](https://mlsanity.com/paper/2301.00858/) (2023)
- [CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning](https://mlsanity.com/paper/2202.07565/) (2022)
- [RLOps: Development Life-cycle of Reinforcement Learning Aided Open RAN](https://mlsanity.com/paper/2111.06978/) (2021)
- [Robust Constrained Reinforcement Learning](https://mlsanity.com/paper/2209.06866/) (2022)
- [Distributionally Robust Constrained Reinforcement Learning under Strong Duality](https://mlsanity.com/paper/2406.15788/) (2024)
- [Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form](https://mlsanity.com/paper/2408.16286/) (2024)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [Explicit Explore, Exploit, or Escape (E4): near-optimal safety-constrained reinforcement learning in polynomial time](https://mlsanity.com/paper/2111.07395/) (2021)
- [A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees](https://mlsanity.com/paper/2401.17780/) (2024)
- [Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality](https://mlsanity.com/paper/2508.17448/) (2025)
- [Sim2real for Reinforcement Learning Driven Next Generation Networks](https://mlsanity.com/paper/2206.03846/) (2022)
- [Robust Lagrangian and Adversarial Policy Gradient for Robust Constrained Markov Decision Processes](https://mlsanity.com/paper/2308.11267/) (2023)
- [Policy Gradient for Robust Markov Decision Processes](https://mlsanity.com/paper/2410.22114/) (2024)
- [A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics](https://mlsanity.com/paper/2505.12583/) (2025)
- [Robust Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2606.00270/) (2026)
- [Generalization in Monitored Markov Decision Processes (Mon-MDPs)](https://mlsanity.com/paper/2505.08988/) (2025)
- [Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPs](https://mlsanity.com/paper/2511.05758/) (2025)
- [Initial Distribution Sensitivity of Constrained Markov Decision Processes](https://mlsanity.com/paper/2510.00348/) (2025)
- [Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees](https://mlsanity.com/paper/2604.14243/) (2026)

## Abstract (arXiv)

In this paper, we focus on the problem of robustifying reinforcement learning (RL) algorithms with respect to model uncertainties. Indeed, in the framework of model-based RL, we propose to merge the theory of constrained Markov decision process (CMDP), with the theory of robust Markov decision process (RMDP), leading to a formulation of robust constrained-MDPs (RCMDP). This formulation, simple in essence, allows us to design RL algorithms that are robust in performance, and provides constraint satisfaction guarantees, with respect to uncertainties in the system's states transition probabilities. The need for RCMPDs is important for real-life applications of RL. For instance, such formulation can play an important role for policy transfer from simulation to real world (Sim2Real) in safety critical applications, which would benefit from performance and safety guarantees which are robust w.r.t model uncertainty. We first propose the general problem formulation under the concept of RCMDP, and then propose a Lagrangian formulation of the optimal problem, leading to a robust-constrained policy gradient RL algorithm. We finally validate this concept on the inventory management problem.