mlsanityData to 14 Oct 2026

A Survey of Constraint Formulations in Safe Reinforcement Learning

arXiv 2402.02025 ↗First posted 3 Feb 2024cs.LG cs.AIcited by 44

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent's policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 57 of its 83 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

57 of 83 references.

References · 57

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2017Safe Reinforcement Learning via Shieldingcited by 449
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  10. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  11. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  12. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
38 more
  1. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  2. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  3. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 287
  4. 2018Safe Exploration in Continuous Action Spacescited by 247
  5. 2019Batch Policy Learning under Constraintscited by 244
  6. 2020Projection-Based Constrained Policy Optimizationcited by 203
  7. 2021Reinforcement learning based recommender systems: A surveycited by 188
  8. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  9. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  10. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  11. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  12. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  13. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  14. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  15. 2020Conservative Safety Critics for Explorationcited by 113
  16. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  17. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  18. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  19. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 75
  20. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 74
  21. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  22. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  23. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 70
  24. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  25. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  26. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  27. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  28. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  29. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  30. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  31. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
  32. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  33. 2019Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamicscited by 47
  34. 2020Cautious Reinforcement Learning with Logical Constraintscited by 45
  35. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 35
  36. 2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 35
  37. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 30
  38. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 28

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 19.

44 papers.

Sort:

By area:

  1. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 15
  2. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 13
  3. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 13
  4. 2024A Review of Safe Reinforcement Learning Methods for Modern Power Systemscited by 11
  5. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 11
  6. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  7. 2024Efficient Policy Evaluation with Safety Constraint for Reinforcement Learningcited by 6
  8. 2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agentscited by 6
  9. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  10. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  11. 2024A Retention-Centric Framework for Continual Learning with Guaranteed Model Developmental Safetycited by 4
  12. 2025Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategiescited by 4
32 more
  1. 2025Universal Dynamic Regret and Constraint Violation Bounds for Constrained Online Convex Optimizationcited by 4
  2. 2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
  3. 2025Safe In-Context Reinforcement Learningcited by 3
  4. 2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
  5. 2025Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signalscited by 2
  6. 2026Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Artcited by 2
  7. 2026CRAX: Fast Safe Reinforcement Learning Benchmarkingcited by 2
  8. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  9. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
  10. 2025Constrained Online Decision-Making: A Unified Frameworkcited by 1
  11. 2025Coloring Between the Lines: Personalization in the Null Space of Planning Constraintscited by 1
  12. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  13. 2026Safe Heterogeneous Multi-Agent RL with Communication Regularization for Coordinated Target Acquisitioncited by 1
  14. 2026How Clinicians Think and What AI Can Learn From Itcited by 1
  15. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfactioncited by 0
  16. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  17. 2025Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulationcited by 0
  18. 2025Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgridscited by 0
  19. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learningcited by 0
  20. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraintscited by 0
  21. 2026A Safety-Constrained Reinforcement Learning Framework for Reliable Wireless Autonomycited by 0
  22. 2026SAGE-LLM: Towards Safe and Generalizable LLM Controller with Fuzzy-CBF Verification and Graph-Structured Knowledge Retrieval for UAV Decisioncited by 0
  23. 2026MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systemscited by 0
  24. 2026Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashingcited by 0
  25. 2026From Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learningcited by 0
  26. 2026Breaking the Epistemic Trap: Active Perception Under Compound Uncertaintycited by 0
  27. 2026Completion vs Optimality: Policy Gradient in Long-Horizon Cumulative-Damage Problemscited by 0
  28. 2026Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deploymentcited by 0
  29. 2026Controller-Augmented Hidden Markov Models: A Computational Framework for Constrained Sequential Inferencecited by 0
  30. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  31. 2026SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventionscited by 0
  32. 2026Learning Chance-Constrained MDPs with Bellman Distributional Certificatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  2. 2024Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning
  3. 2024Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning
  4. 2024Safe and Robust Reinforcement Learning: Principles and Practice
  5. 2025Safety Representations for Safer Policy Learning
  6. 2026Evaluation Metrics for Safe Reinforcement Learning
  7. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learning
  8. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning