Verified Safe Reinforcement Learning for Neural Network Dynamic Models

arXiv 2405.15994 ↗First posted 25 May 2024cs.LG cs.AIcited by 7

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Learning reliably safe autonomous control is one of the core problems in trustworthy autonomy. However, training a controller that can be formally verified to be safe remains a major challenge. We introduce a novel approach for learning verified safe control policies in nonlinear neural dynamical systems while maximizing overall performance. Our approach aims to achieve safety in the sense of finite-horizon reachability proofs, and is comprised of three key parts. The first is a novel curriculum learning scheme that iteratively increases the verified safe horizon. The second leverages the iterative nature of gradient-based learning to leverage incremental verification, reusing information from prior verification runs. Finally, we learn multiple verified initial-state-dependent controllers, an idea that is especially valuable for more complex domains where learning a single universal verified safe controller is extremely challenging. Our experiments on five safe control problems demonstrate that our trained controllers can achieve verified safety over horizons that are as much as an order of magnitude longer than state-of-the-art baselines, while maintaining high reward, as well as a perfect safety record over entire episodes. Our code is available at https://github.com/jlwu002/VSRL.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 58 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

37 of 58 references.

References · 37

2017Constrained Policy Optimizationcited by 983
2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
2017Safe Reinforcement Learning via Shieldingcited by 449
2018Efficient Neural Network Robustness Certification with General Activation Functionscited by 445
2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 328
2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
2016Safety Verification of Deep Neural Networkscited by 263
2020Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifierscited by 187
2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181

25 more

2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
2018A Dual Approach to Scalable Verification of Deep Networkscited by 125
2022General Cutting Planes for Bound-Propagation-Based Neural Network Verificationcited by 103
2021Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functionscited by 94
2022Complete Verification via Multi-Neuron Relaxation Guided Branch-and-Boundcited by 92
2018Verisig: verifying safety properties of hybrid systems with neural network controllerscited by 84
2021Lyapunov-stable neural-network controlcited by 66
2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 59
2022Reachability Constrained Reinforcement Learningcited by 59
2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
2018Linear model predictive safety certification for learning-based controlcited by 37
2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 37
2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 29
2022Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopescited by 26
2023Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learningcited by 24
2021Safe Control with Neural Network Dynamic Modelscited by 22
2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 16
2023Model-Based Control with Sparse Neural Dynamicscited by 16
2022Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculumcited by 15
2019Verification of Non-Linear Specifications for Neural Networkscited by 14
2022Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programmingcited by 12
2023POLAR-Express: Efficient and Precise Formal Reachability Analysis of Neural-Network Controlled Systemscited by 12
2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 11
2019Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithmcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

2026Parallel Differentiable Reachability for Learning and Planning with Certified Neural Dynamics and Controllerscited by 2
2025Learning Vision-Based Neural Network Controllers with Semi-Probabilistic Safety Guaranteescited by 1
2025Learning Verifiable Control Policies Using Relaxed Verificationcited by 1
2025A Dynamical Systems Framework for Reinforcement Learning Safety and Robustness Verificationcited by 0
2026Conformal Reachability for Safe Control in Unknown Environmentscited by 0
2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
2026COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenancecited by 0

Similar, not cited

Close in wording, no citation either way