Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programming
Not on the map. Why?
Abstract (arXiv)
Safety assurance is uncompromisable for safety-critical environments with the presence of drastic model uncertainties (e.g., distributional shift), especially with humans in the loop. However, incorporating uncertainty in safe learning will naturally lead to a bi-level problem, where at the lower level the (worst-case) safety constraint is evaluated within the uncertainty ambiguity set. In this paper, we present a tractable distributionally safe reinforcement learning framework to enforce safety under a distributional shift measured by a Wasserstein metric. To improve the tractability, we first use duality theory to transform the lower-level optimization from infinite-dimensional probability space where distributional shift is measured, to a finite-dimensional parametric space. Moreover, by differentiable convex programming, the bi-level safe learning problem is further reduced to a single-level one with two sequential computationally efficient modules: a convex quadratic program to guarantee safety followed by a projected gradient ascent to simultaneously find the worst-case uncertainty. This end-to-end differentiable framework with safety constraints, to the best of our knowledge, is the first tractable single-level solution to address distributional safety. We test our approach on first and second-order systems with varying complexities and compare our results with the uncertainty-agnostic policies, where our approach demonstrates a significant improvement on safety guarantees.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 67 references that are in the map. First posted on arXiv 3 Oct 2023.
Builds on
42 of 67 references.
References · 42
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2017Constrained Policy Optimizationcited by 928
- 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
- 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
- 2019Differentiable Convex Optimization Layerscited by 458
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
- 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
- 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
- 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 261
30 more
- 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
- 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
- 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
- 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
- 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
- 2019Efficient Algorithms for Smooth Minimax Optimizationcited by 75
- 2019Differentiating Through a Cone Programcited by 72
- 2020EGAD! an Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulationcited by 61
- 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 61
- 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 44
- 2021Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Gamescited by 40
- 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 38
- 2019On Solving Minimax Optimization Locally: A Follow-the-Ridge Approachcited by 34
- 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
- 2018Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approachcited by 27
- 2021Safe Pontryagin Differentiable Programmingcited by 25
- 2021Meta-Adaptive Nonlinear Control: Theory and Algorithmscited by 23
- 2021Distributionally Robust Policy Learning via Adversarial Environment Generationcited by 17
- 2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
- 2020Distributionally Robust Chance Constrained Data-enabled Predictive Controlcited by 13
- 2022No-Regret Learning in Dynamic Stackelberg Gamescited by 12
- 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
- 2020Wasserstein Distributionally Robust Motion Control for Collision Avoidance Using Conditional Value-at-Riskcited by 8
- 2020Safe Reinforcement Learning of Control-Affine Systems with Vertex Networkscited by 8
- 2022Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variablescited by 6
- 2020FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimizecited by 5
- 2021Data-driven distributionally robust MPC for constrained stochastic systemscited by 3
- 2023Robust Control Barrier Functions with Uncertainty Estimationcited by 3
- 2023On the Optimality, Stability, and Feasibility of Control Barrier Functions: An Adaptive Learning-Based Approachcited by 3
- 2023Wasserstein Distributionally Robust Control Barrier Function using Conditional Value-at-Risk with Differentiable Convex Programmingcited by 2
Built on by
Citing papers per year: 2023: 1.
1 paper.