# A Review of Safe Reinforcement Learning: Methods, Theory and Applications

arXiv: https://arxiv.org/abs/2205.10330 · HTML: https://mlsanity.com/paper/2205.10330/

## Key facts

- Cited by 181 papers in the map (data through 14 Oct 2026).
- Builds on 141 of its 323 references that are in the map.
- First posted on arXiv 20 May 2022.

## Builds on (most cited first, 30 of 141)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Recent Advances in Convolutional Neural Networks](https://mlsanity.com/paper/1512.07108/) (2015)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)

## Built on by (most cited first, 30 of 181)

- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2310.12567/) (2023)
- [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research](https://mlsanity.com/paper/2305.09304/) (2023)
- [SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning](https://mlsanity.com/paper/2503.03480/) (2025)
- [Constrained Decision Transformer for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2302.07351/) (2023)
- [Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents](https://mlsanity.com/paper/2309.09919/) (2023)
- [Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model](https://mlsanity.com/paper/2401.10700/) (2024)
- [Datasets and Benchmarks for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2306.09303/) (2023)
- [State-wise Safe Reinforcement Learning: A Survey](https://mlsanity.com/paper/2302.03122/) (2023)
- [Enhancing LLM Safety via Constrained Direct Preference Optimization](https://mlsanity.com/paper/2403.02475/) (2024)
- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [On the Robustness of Safe Reinforcement Learning under Observational Perturbations](https://mlsanity.com/paper/2205.14691/) (2022)
- [Iterative Reachability Estimation for Safe Reinforcement Learning](https://mlsanity.com/paper/2309.13528/) (2023)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Safe Offline Reinforcement Learning with Real-Time Budget Constraints](https://mlsanity.com/paper/2306.00603/) (2023)
- [Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs](https://mlsanity.com/paper/2306.11700/) (2023)
- [Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning](https://mlsanity.com/paper/2306.12755/) (2023)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations](https://mlsanity.com/paper/2509.20478/) (2025)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints](https://mlsanity.com/paper/2301.10923/) (2023)
- [Doubly Mild Generalization for Offline Reinforcement Learning](https://mlsanity.com/paper/2411.07934/) (2024)
- [Diffusion-Reward Adversarial Imitation Learning](https://mlsanity.com/paper/2405.16194/) (2024)
- [SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents](https://mlsanity.com/paper/2405.20539/) (2024)
- [Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning](https://mlsanity.com/paper/2310.03718/) (2023)
- [Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression](https://mlsanity.com/paper/2410.19400/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) has achieved tremendous success in many complex decision-making tasks. However, safety concerns are raised during deploying RL in real-world applications, leading to a growing demand for safe RL algorithms, such as in autonomous driving and robotics scenarios. While safe control has a long history, the study of safe RL algorithms is still in the early stages. To establish a good foundation for future safe RL research, in this paper, we provide a review of safe RL from the perspectives of methods, theories, and applications. Firstly, we review the progress of safe RL from five dimensions and come up with five crucial problems for safe RL being deployed in real-world applications, coined as "2H3W". Secondly, we analyze the algorithm and theory progress from the perspectives of answering the "2H3W" problems. Particularly, the sample complexity of safe RL algorithms is reviewed and discussed, followed by an introduction to the applications and benchmarks of safe RL algorithms. Finally, we open the discussion of the challenging problems in safe RL, hoping to inspire future research on this thread. To advance the study of safe RL algorithms, we release an open-sourced repository containing the implementations of major safe RL algorithms at the link: https://github.com/chauncygu/Safe-Reinforcement-Learning-Baselines.git.