mlsanityData to 14 Oct 2026

The Feasibility Theory of Constrained Reinforcement Learning: A Tutorial Study

arXiv 2404.10064 ↗First posted 15 Apr 2024eess.SY cs.SYcited by 1

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Satisfying safety constraints is a priority concern when solving optimal control problems (OCPs). Due to the existence of infeasibility phenomenon, where a constraint-satisfying solution cannot be found, it is necessary to identify a feasible region before implementing a policy. Existing feasibility theories built for model predictive control (MPC) only consider the feasibility of optimal policy. However, reinforcement learning (RL), as another important control method, solves the optimal policy in an iterative manner, which comes with a series of non-optimal intermediate policies. Feasibility analysis of these non-optimal policies is also necessary for iteratively improving constraint satisfaction; but that is not available under existing MPC feasibility theories. This paper proposes a feasibility theory that applies to both MPC and RL by filling in the missing part of feasibility analysis for an arbitrary policy. The basis of our theory is to decouple policy solving and implementation into two temporal domains: virtual-time domain and real-time domain. This allows us to separately define initial and endless, state and policy feasibility, and their corresponding feasible regions. Based on these definitions, we analyze the containment relationships between different feasible regions, which enables us to describe the feasible region of an arbitrary policy. We further provide virtual-time constraint design rules along with a practical design tool called feasibility function that helps to achieve the maximum feasible region. We review most of existing constraint formulations and point out that they are essentially applications of feasibility functions in different forms. We demonstrate our feasibility theory by visualizing different feasible regions under both MPC and RL policies in an emergency braking control task.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 71 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

41 of 71 references.

References · 41

  1. 2017Constrained Policy Optimizationcited by 983
  2. 2019Control Barrier Functions: Theory and Applicationscited by 372
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  4. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  6. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  7. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  8. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 287
  9. 2020Projection-Based Constrained Policy Optimizationcited by 203
  10. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 201
  11. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  12. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
29 more
  1. 2020Learning Control Barrier Functions from Expert Demonstrationscited by 92
  2. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 91
  3. 2021Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificatescited by 74
  4. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  5. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  6. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  7. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  8. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 59
  9. 2022Reachability Constrained Reinforcement Learningcited by 59
  10. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  11. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  12. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 34
  13. 2021Model-based Constrained Reinforcement Learning using Generalized Control Barrier Functioncited by 32
  14. 2019Control Barrier Functions for Systems with High Relative Degreecited by 31
  15. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 30
  16. 2022Safe Control Under Input Limits with Neural Control Barrier Functionscited by 29
  17. 2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 27
  18. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 27
  19. 2021Safe Control with Neural Network Dynamic Modelscited by 22
  20. 2018Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigationcited by 18
  21. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 18
  22. 2021Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangiancited by 15
  23. 2022Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Modelscited by 15
  24. 2021Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachabilitycited by 14
  25. 2022Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programmingcited by 12
  26. 2022Safety Index Synthesis via Sum-of-Squares Programmingcited by 12
  27. 2018A Classification-based Approach for Approximate Reachabilitycited by 10
  28. 2024Real-time Safety Index Adaptation for Parameter-varying Systems via Determinant Gradient Ascendcited by 4
  29. 2023Safety Index Synthesis with State-dependent Control Spacecited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Verification of Neural Control Barrier Functions with Symbolic Derivative Bounds Propagationcited by 12

Similar, not cited

Close in wording, no citation either way

  1. 2024Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence
  2. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  3. 2025Predictive Lagrangian Optimization for Constrained Reinforcement Learning
  4. 2025Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving
  5. 2026Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning
  6. 2024Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning
  7. 2024Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems
  8. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction