mlsanityData to 7 Oct 2026

Constrained Reinforcement Learning Under Model Mismatch

arXiv 2405.01327 ↗First posted 2 May 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Existing studies on constrained reinforcement learning (RL) may obtain a well-performing policy in the training environment. However, when deployed in a real environment, it may easily violate constraints that were originally satisfied during training because there might be model mismatch between the training and real environments. To address the above challenge, we formulate the problem as constrained RL under model uncertainty, where the goal is to learn a good policy that optimizes the reward and at the same time satisfy the constraint under model mismatch. We develop a Robust Constrained Policy Optimization (RCPO) algorithm, which is the first algorithm that applies to large/continuous state space and has theoretical guarantees on worst-case reward improvement and constraint violation at each iteration during the training. We demonstrate the effectiveness of our algorithm on a set of RL tasks with constraints.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

37 of 54 references.

References · 37

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2016OpenAI Gymcited by 1,684
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  7. 2018Reward Constrained Policy Optimizationcited by 397
  8. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  9. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  10. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  11. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  12. 2018Safe Exploration in Continuous Action Spacescited by 237
25 more
  1. 2020Projection-Based Constrained Policy Optimizationcited by 195
  2. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  3. 2020Exploration-Exploitation in Constrained MDPscited by 146
  4. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  5. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  6. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  7. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  8. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  9. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  10. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  11. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  12. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  13. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  14. 2017Reinforcement Learning under Model Mismatchcited by 52
  15. 2020Constrained Upper Confidence Reinforcement Learningcited by 46
  16. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  17. 2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 34
  18. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  19. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  20. 2023Robust Average-Reward Markov Decision Processescited by 22
  21. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  22. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  23. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
  24. 2020Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecificationcited by 16
  25. 2022Robust Constrained Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  2. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
  3. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  4. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  5. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  6. 2026EvoNash-MARL: A Closed-Loop Multi-Agent Reinforcement Learning Framework for Medium-Horizon Equity Allocationcited by 1