# Safe Continual Reinforcement Learning in Non-stationary Environments

arXiv: https://arxiv.org/abs/2604.19737 · HTML: https://mlsanity.com/paper/2604.19737/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 38 of its 64 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2310.12567/) (2023)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Deep Reinforcement Learning with Plasticity Injection](https://mlsanity.com/paper/2305.15555/) (2023)
- [Safe Reinforcement Learning with Model Uncertainty Estimates](https://mlsanity.com/paper/1810.08700/) (2018)
- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version](https://mlsanity.com/paper/1904.10090/) (2019)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [Continual Reinforcement Learning with Complex Synapses](https://mlsanity.com/paper/1802.07239/) (2018)
- [Towards Safe Policy Improvement for Non-Stationary MDPs](https://mlsanity.com/paper/2010.12645/) (2020)
- [Safe Reinforcement Learning Using Black-Box Reachability Analysis](https://mlsanity.com/paper/2204.07417/) (2022)
- [A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/2405.19153/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) offers a compelling data-driven paradigm for synthesizing controllers for complex systems when accurate physical models are unavailable; however, most existing control-oriented RL methods assume stationarity and, therefore, struggle in real-world non-stationary deployments where system dynamics and operating conditions can change unexpectedly. Moreover, RL controllers acting in physical environments must satisfy safety constraints throughout their learning and execution phases, rendering transient violations during adaptation unacceptable. Although continual RL and safe RL have each addressed non-stationarity and safety, respectively, their intersection remains comparatively unexplored, motivating the study of safe continual RL algorithms that can adapt over the system's lifetime while preserving safety. In this work, we systematically investigate safe continual reinforcement learning by introducing three benchmark environments that capture safety-critical continual adaptation and by evaluating representative approaches from safe RL, continual RL, and their combinations. Our empirical results reveal a fundamental tension between maintaining safety constraints and preventing catastrophic forgetting under non-stationary dynamics, with existing methods generally failing to achieve both objectives simultaneously. To address this shortcoming, we examine regularization-based strategies that partially mitigate this trade-off and characterize their benefits and limitations. Finally, we outline key open challenges and research directions toward developing safe, resilient learning-based controllers capable of sustained autonomous operation in changing environments.