mlsanityData to 14 Oct 2026

A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety

arXiv 2505.17342 ↗First posted 22 May 2025cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Safe Reinforcement Learning (SafeRL) is the subfield of reinforcement learning that explicitly deals with safety constraints during the learning and deployment of agents. This survey provides a mathematically rigorous overview of SafeRL formulations based on Constrained Markov Decision Processes (CMDPs) and extensions to Multi-Agent Safe RL (SafeMARL). We review theoretical foundations of CMDPs, covering definitions, constrained optimization techniques, and fundamental theorems. We then summarize state-of-the-art algorithms in SafeRL for single agents, including policy gradient methods with safety guarantees and safe exploration strategies, as well as recent advances in SafeMARL for cooperative and competitive settings. Additionally, we propose five open research problems to advance the field, with three focusing on SafeMARL. Each problem is described with motivation, key challenges, and related prior work. This survey is intended as a technical guide for researchers interested in SafeRL and SafeMARL, highlighting key concepts, methods, and open future research directions.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 53 of its 120 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

53 of 120 references.

References · 53

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  6. 2017Safe Reinforcement Learning via Shieldingcited by 449
  7. 2018Reward Constrained Policy Optimizationcited by 417
  8. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  9. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  11. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  12. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
38 more
  1. 2018Safe Exploration in Continuous Action Spacescited by 247
  2. 2020Projection-Based Constrained Policy Optimizationcited by 203
  3. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  4. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  5. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  6. 2017On a Formal Model of Safe and Scalable Self-driving Carscited by 131
  7. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  8. 2017AI Safety Gridworldscited by 110
  9. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  10. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  11. 2015Safe Controller Optimization for Quadrotors with Gaussian Processescited by 78
  12. 2015Policy Gradient for Coherent Risk Measurescited by 76
  13. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 61
  14. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 50
  15. 2021safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Roboticscited by 50
  16. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  17. 2018Linear model predictive safety certification for learning-based controlcited by 37
  18. 2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
  19. 2016Distributed Constraint Optimization Problems and Applications: A Surveycited by 26
  20. 2014Policy Gradients for CVaR-Constrained MDPscited by 21
  21. 2018Model-Based Reinforcement Learning for Sepsis Treatmentcited by 21
  22. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  23. 2017Safety-Aware Apprenticeship Learningcited by 12
  24. 2018Learning Safe Policies with Expert Guidancecited by 11
  25. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learningcited by 9
  26. 2024Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Drivingcited by 9
  27. 2019SafeLife 1.0: Exploring Side Effects in Complex Environmentscited by 8
  28. 2021Barrier Function-based Safe Reinforcement Learning for Emergency Control of Power Systemscited by 8
  29. 2013Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Methodcited by 5
  30. 2024Safety through feedback in Constrained RLcited by 5
  31. 2020Safe Reinforcement Learning through Meta-learned Instinctscited by 4
  32. 2017Constraint Coupled Distributed Optimization: a Relaxation and Duality Approachcited by 3
  33. 2014Deadline Differentiated Pricing of Deferrable Electric Loadscited by 2
  34. 2022Safe Equilibriumcited by 2
  35. 2023A Tutorial on Distributed Optimization for Cooperative Robotics: from Setups and Algorithms to Toolboxes and Research Directionscited by 2
  36. 2025On the Design of Safe Continual RL Methods for Control of Nonlinear Systemscited by 2
  37. 2016Distributed Frequency Control in Power Grids Under Limited Communicationcited by 1
  38. 2017Distributed Optimal Frequency Control Considering a Nonlinear Network-Preserving Modelcited by 1

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  2. 2026Safe Multi-Agent Behavior Must Be Maintained, Not Merely Asserted: Constraint Drift in LLM-Based Multi-Agent Systemscited by 3
  3. 2025Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signalscited by 2
  4. 2026Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Surveycited by 1
  5. 2026Safety-Critical Reinforcement Learning with Viability-Based Action Shielding for Hypersonic Longitudinal Flightcited by 0
  6. 2026A Safety-Constrained Reinforcement Learning Framework for Reliable Wireless Autonomycited by 0
  7. 2026Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deploymentcited by 0
  8. 2026Coordination Graphs for Constrained Multi-Agent Reinforcement Learningcited by 0
  9. 2026Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policiescited by 0
  10. 2026Controller-Augmented Hidden Markov Models: A Computational Framework for Constrained Sequential Inferencecited by 0
  11. 2026Bias-Controlled Primal-Dual Natural Actor-Critic: Optimal Rates for Constrained Multi-Objective Average-Reward RLcited by 0
  12. 2026Lightweight Safe Reinforcement Learning for End-to-End UAV Navigationcited by 0
1 more
  1. 2026Safe Greenhouse Climate Control Using Lagrangian-Constrained PPO with Kolmogorov-Arnold Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium
  2. 2026Evaluation Metrics for Safe Reinforcement Learning
  3. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
  4. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applications
  5. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  6. 2025Probabilistic Shielding for Safe Reinforcement Learning
  7. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  8. 2024Safe and Robust Reinforcement Learning: Principles and Practice