Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints

arXiv 2605.18842 ↗First posted 13 May 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safe reinforcement learning in nonstationary environments requires safety mechanisms that adapt as environmental conditions change. Standard safe reinforcement learning methods often assume fixed constraints or stable environmental conditions, which can become inadequate under distribution shift. We propose LILAC+, a framework for safe continual reinforcement learning under nonstationarity that combines three adaptive safety mechanisms: context-based safety constraints, adaptation-speed constraints, and budget-to-state safety enforcement. Context-based constraints adjust safety requirements using inferred and predicted environmental context. Adaptation-speed constraints tighten safety requirements when the rate of environmental change exceeds the agent's ability to adapt safely. Budget-to-state enforcement converts cumulative safety requirements into local state-level control constraints that can be enforced at decision time. Together, these mechanisms provide a unified approach for proactive and reactive safety adaptation in continual reinforcement learning. We evaluate the framework in simulated driving environments under stationary, seen nonstationary, and unseen nonstationary conditions. The results show that adaptive safety constraints substantially reduce safety violations under distribution shift while maintaining competitive task performance compared with unconstrained and fixed-constraint baselines. These findings suggest that safe continual reinforcement learning requires adaptive constraint mechanisms that respond not only to current state information but also to predicted environmental context, adaptation demand, and remaining safety budget.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 26 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

16 of 26 references.

References · 16

2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
2017Constrained Policy Optimizationcited by 983
2017Safe Reinforcement Learning via Shieldingcited by 449
2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
2015Policy Gradient for Coherent Risk Measurescited by 76
2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 68
2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 56
2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 43
2020Optimizing for the Future in Non-Stationary MDPscited by 37

4 more

2022Robust Multi-Objective Bayesian Optimization Under Input Noisecited by 37
2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
2023Provably Efficient Model-Free Algorithms for Non-stationary CMDPscited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way