# Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environments

arXiv: https://arxiv.org/abs/2303.14265 · HTML: https://mlsanity.com/paper/2303.14265/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 11 of 11)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Cautious Reinforcement Learning with Logical Constraints](https://mlsanity.com/paper/2002.12156/) (2020)
- [Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function](https://mlsanity.com/paper/2103.01556/) (2021)
- [Deep Q-learning from Demonstrations](https://mlsanity.com/paper/1704.03732/) (2017)
- [Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions](https://mlsanity.com/paper/1908.01883/) (2019)

## Built on by (most cited first, 4 of 4)

- [Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation](https://mlsanity.com/paper/2405.20860/) (2024)
- [Safe Reinforcement Learning for Power System Control: A Review](https://mlsanity.com/paper/2407.00681/) (2024)
- [Modeling Risk in Reinforcement Learning: A Literature Mapping](https://mlsanity.com/paper/2312.05231/) (2023)
- [REVEAL-IT: REinforcement learning with Visibility of Evolving Agent poLicy for InTerpretability](https://mlsanity.com/paper/2406.14214/) (2024)

## Abstract (arXiv)

This study proposes a safe and sample-efficient reinforcement learning (RL) framework to address two major challenges in developing applicable RL algorithms: satisfying safety constraints and efficiently learning with limited samples. To guarantee safety in real-world complex environments, we use the safe set algorithm (SSA) to monitor and modify the nominal controls, and evaluate SSA+RL in a clustered dynamic environment which is challenging to be solved by existing RL algorithms. However, the SSA+RL framework is usually not sample-efficient especially in reward-sparse environments, which has not been addressed in previous safe RL works. To improve the learning efficiency, we propose three techniques: (1) avoiding behaving overly conservative by adapting the SSA; (2) encouraging safe exploration using random network distillation with safety constraints; (3) improving policy convergence by treating SSA as expert demonstrations and directly learn from that. The experimental results show that our framework can achieve better safety performance compare to other safe RL methods during training and solve the task with substantially fewer episodes. Project website: https://hychen-naza.github.io/projects/Safe_RL/.