A Provable Approach for End-to-End Safe Reinforcement Learning

arXiv 2505.21852 ↗First posted 28 May 2025cs.LG cs.AI cs.ITcited by 2

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 61 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

46 of 61 references.

Closest ancestors · 1

Same direction, newest first

2024A Survey of Constraint Formulations in Safe Reinforcement Learning3 Feb

Other references · 45

2022Training language models to follow instructions with human feedbackcited by 15,971
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2016Concrete Problems in AI Safetycited by 1,642
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2017Constrained Policy Optimizationcited by 983
2023Training Diffusion Models with Reinforcement Learningcited by 982
2015End-to-End Training of Deep Visuomotor Policiescited by 875
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
2017Safe Reinforcement Learning via Shieldingcited by 449
2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
2017On Kernelized Multi-armed Banditscited by 352

33 more

2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
2019Reinforcement Learning in Healthcare: A Surveycited by 310
2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
2019Batch Policy Learning under Constraintscited by 244
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
2021Recent Advances in Reinforcement Learning in Financecited by 110
2019Training Agents using Upside-Down Reinforcement Learningcited by 107
2019Reward-Conditioned Policiescited by 102
2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
2021Benchmarks for Deep Off-Policy Evaluationcited by 89
2018Stagewise Safe Bayesian Optimization with Gaussian Processescited by 79
2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 75
2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
2020Verifiably Safe Exploration for End-to-End Reinforcement Learningcited by 34
2022Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarkingcited by 34
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 28
2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 28
2022Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guaranteescited by 27
2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
2017Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learningcited by 18
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18
2023Sample Path Regularity of Gaussian Processes from the Covariance Kernelcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

2026Safe Exploration via Policy Priorscited by 6
2025Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Controlcited by 0

Similar, not cited

Close in wording, no citation either way