mlsanityData to 14 Oct 2026

Robust Constrained Reinforcement Learning

arXiv 2209.06866 ↗First posted 14 Sep 2022cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Constrained reinforcement learning is to maximize the expected reward subject to constraints on utilities/costs. However, the training environment may not be the same as the test one, due to, e.g., modeling error, adversarial attack, non-stationarity, resulting in severe performance degradation and more importantly constraint violation. We propose a framework of robust constrained reinforcement learning under model uncertainty, where the MDP is not fixed but lies in some uncertainty set, the goal is to guarantee that constraints on utilities/costs are satisfied for all MDPs in the uncertainty set, and to maximize the worst-case reward performance over the uncertainty set. We design a robust primal-dual approach, and further theoretically develop guarantee on its convergence, complexity and robust feasibility. We then investigate a concrete example of δ-contamination uncertainty set, design an online and model-free algorithm and theoretically characterize its sample complexity.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

50 of 80 references.

References · 50

  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  6. 2017Robust Adversarial Reinforcement Learningcited by 451
  7. 2018Reward Constrained Policy Optimizationcited by 417
  8. 2017Adversarial Attacks on Neural Network Policiescited by 320
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  10. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  11. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  12. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
38 more
  1. 2018Safe Exploration in Continuous Action Spacescited by 247
  2. 2020Projection-Based Constrained Policy Optimizationcited by 203
  3. 2018Generative Adversarial Imitation from Observationcited by 184
  4. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  5. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  6. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  7. 2020Exploration-Exploitation in Constrained MDPscited by 149
  8. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  9. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  10. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  11. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  12. 2017Delving into adversarial attacks on deep policiescited by 104
  13. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  14. 2017Implicit Regularization in Deep Learningcited by 90
  15. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  16. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  17. 2018Robust Estimation via Robust Gradient Estimationcited by 76
  18. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 75
  19. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  20. 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
  21. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  22. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  23. 2020A Unified Single-loop Alternating Gradient Projection Algorithm for Nonconvex-Concave and Convex-Nonconcave Minimax Problemscited by 57
  24. 2019Wasserstein Robust Reinforcement Learningcited by 54
  25. 2017Reinforcement Learning under Model Mismatchcited by 52
  26. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 50
  27. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  28. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 34
  29. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 33
  30. 2020Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertaintycited by 23
  31. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 21
  32. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 19
  33. 2020Finite-sample Analysis of Greedy-GQ with Linear Function Approximation under Markovian Noisecited by 17
  34. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  35. 2020Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecificationcited by 16
  36. 2022Robust Learning from Observation with Model Misspecificationcited by 9
  37. 2018Robust Nonparametric Regression under Huber's ε-contamination Modelcited by 6
  38. 2021Lyapunov Robust Constrained-MDPs: Soft-Constrained Robustly Stable Policy Optimization under Model Uncertaintycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  2. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
  3. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  4. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  5. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 8
  6. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 8
  7. 2024A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guaranteescited by 6
  8. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  9. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  10. 2025Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPscited by 1
  11. 2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
  12. 2025Transition Uncertainties in Constrained Markov Decision Models: A Robust Optimization Approachcited by 0
3 more
  1. 2025Initial Distribution Sensitivity of Constrained Markov Decision Processescited by 0
  2. 2026Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guaranteescited by 0
  3. 2026Robust Peak-cost Constrained Reinforcement Learningcited by 0