# Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret

arXiv: https://arxiv.org/abs/1505.05798 · HTML: https://mlsanity.com/paper/1505.05798/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2015.

## Built on by (most cited first, 15 of 15)

- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interaction](https://mlsanity.com/paper/2209.13308/) (2022)
- [Constrained Reinforcement Learning with Smoothed Log Barrier Function](https://mlsanity.com/paper/2403.14508/) (2024)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Safe Reinforcement Learning using Data-Driven Predictive Control](https://mlsanity.com/paper/2211.11027/) (2022)
- [Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning](https://mlsanity.com/paper/2409.12045/) (2024)
- [Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2206.00270/) (2022)
- [Off-Policy Evaluation for Action-Dependent Non-Stationary Environments](https://mlsanity.com/paper/2301.10330/) (2023)
- [On the Design of Safe Continual RL Methods for Control of Nonlinear Systems](https://mlsanity.com/paper/2502.15922/) (2025)
- [Constrained Reinforcement Learning for Safe Heat Pump Control](https://mlsanity.com/paper/2409.19716/) (2024)
- [Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey](https://mlsanity.com/paper/2111.14324/) (2021)
- [Revisiting Safe Exploration in Safe Reinforcement learning](https://mlsanity.com/paper/2409.01245/) (2024)
- [SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning](https://mlsanity.com/paper/2604.09452/) (2026)
- [Safe Continual Reinforcement Learning in Non-stationary Environments](https://mlsanity.com/paper/2604.19737/) (2026)

## Abstract (arXiv)

Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial set- ting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control.