# Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery

arXiv: https://arxiv.org/abs/2306.13944 · HTML: https://mlsanity.com/paper/2306.13944/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 24 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Safe Reinforcement Learning in Constrained Markov Decision Processes](https://mlsanity.com/paper/2008.06626/) (2020)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Safe Reinforcement Learning by Imagining the Near Future](https://mlsanity.com/paper/2202.07789/) (2022)
- [State-wise Safe Reinforcement Learning: A Survey](https://mlsanity.com/paper/2302.03122/) (2023)
- [Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping](https://mlsanity.com/paper/1910.02787/) (2019)
- [Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Control](https://mlsanity.com/paper/2011.08421/) (2020)
- [Medical Dead-ends and Learning to Identify High-risk States and Treatments](https://mlsanity.com/paper/2110.04186/) (2021)
- [Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2101.09458/) (2021)
- [MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance](https://mlsanity.com/paper/2112.03575/) (2021)
- [Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning](https://mlsanity.com/paper/2301.05664/) (2023)
- [Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning](https://mlsanity.com/paper/2212.06998/) (2022)

## Built on by (most cited first, 5 of 5)

- [DPMambaIR: All-in-One Image Restoration via Degradation-Aware Prompt State Space Model](https://mlsanity.com/paper/2504.17732/) (2025)
- [Towards Efficient Constraint Handling in Neural Solvers for Routing Problems](https://mlsanity.com/paper/2602.16012/) (2026)
- [Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2405.12001/) (2024)
- [Constraint-Aware Reinforcement Learning via Adaptive Action Scaling](https://mlsanity.com/paper/2510.11491/) (2025)
- [Reversible Simplex Supervision with Post-Action Debt Accounting for Goal-Reaching RL](https://mlsanity.com/paper/2601.19499/) (2026)

## Abstract (arXiv)

Safety is one of the main challenges in applying reinforcement learning to realistic environmental tasks. To ensure safety during and after training process, existing methods tend to adopt overly conservative policy to avoid unsafe situations. However, overly conservative policy severely hinders the exploration, and makes the algorithms substantially less rewarding. In this paper, we propose a method to construct a boundary that discriminates safe and unsafe states. The boundary we construct is equivalent to distinguishing dead-end states, indicating the maximum extent to which safe exploration is guaranteed, and thus has minimum limitation on exploration. Similar to Recovery Reinforcement Learning, we utilize a decoupled RL framework to learn two policies, (1) a task policy that only considers improving the task performance, and (2) a recovery policy that maximizes safety. The recovery policy and a corresponding safety critic are pretrained on an offline dataset, in which the safety critic evaluates upper bound of safety in each state as awareness of environmental safety for the agent. During online training, a behavior correction mechanism is adopted, ensuring the agent to interact with the environment using safe actions only. Finally, experiments of continuous control tasks demonstrate that our approach has better task performance with less safety violations than state-of-the-art algorithms.