Leveraging Analytic Gradients in Provably Safe Reinforcement Learning

arXiv 2506.01665 ↗First posted 2 Jun 2025cs.LG cs.AI cs.ROcited by 1

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

The deployment of autonomous robots in safety-critical applications requires safety guarantees. Provably safe reinforcement learning is an active field of research that aims to provide such guarantees using safeguards. These safeguards should be integrated during training to reduce the sim-to-real gap. While there are several approaches for safeguarding sampling-based reinforcement learning, analytic gradient-based reinforcement learning often achieves superior performance from fewer environment interactions. However, there is no safeguarding approach for this learning paradigm yet. Our work addresses this gap by developing the first effective safeguard for analytic gradient-based reinforcement learning. We analyse existing, differentiable safeguards, adapt them through modified mappings and gradient formulations, and integrate them into a state-of-the-art learning algorithm and a differentiable simulation. Using numerical experiments on three control tasks, we evaluate how different safeguards affect learning. The results demonstrate safeguarded training without compromising performance. Additional visuals are provided at timwalter.github.io/safe-agb-rl.github.io.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 47 of its 90 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

47 of 90 references.

Closest ancestors · 1

Same direction, newest first

2024Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents15 Oct

Other references · 46

2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2017Proximal Policy Optimization Algorithmscited by 15,215
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
2019Differentiable Convex Optimization Layerscited by 465
2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328

34 more

2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
2017A Rewriting System for Convex Optimization Problemscited by 179
2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
2021Core Challenges of Social Robot Navigation: A Surveycited by 106
2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
2024Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Controlcited by 99
2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
2019Differentiating Through a Cone Programcited by 73
2021Physics-based Deep Learningcited by 68
2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 63
2020Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?cited by 34
2022Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarkingcited by 34
2024Learning Quadruped Locomotion Using Differentiable Simulationcited by 32
2019Learning Convex Optimization Control Policiescited by 31
2021Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimizationcited by 31
2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 26
2022Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopescited by 26
2021Computationally Efficient Safe Reinforcement Learning for Power Systemscited by 20
2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulationcited by 20
2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 20
2024Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Maskingcited by 19
2024Learning Quadrotor Control From Visual Features Using Differentiable Simulationcited by 19
2022Provably Safe Deep Reinforcement Learning for Robotic Manipulation in Human Environmentscited by 18
2020Learning Convex Optimization Modelscited by 16
2022Contingency-constrained economic dispatch with safe reinforcement learningcited by 8
2019Linear Encodings for Polytope Containment Problemscited by 6
2021Guarantees for Real Robotic Systems: Unifying Formal Controller Synthesis and Reachset-Conformant Identificationcited by 5
2022Safe Reinforcement Learning using Data-Driven Predictive Controlcited by 5
2024Reachset-Conformant System Identificationcited by 5
2024CuClarabel: GPU Acceleration for a Conic Optimization Solvercited by 4
2022Embedded Code Generation with CVXPYcited by 3
2023Safe Neural Control for Non-Affine Control Systems with Differentiable Control Barrier Functionscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

2025Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?cited by 5

Similar, not cited

Close in wording, no citation either way