# Physics-model-guided Worst-case Sampling for Safe Reinforcement Learning

arXiv: https://arxiv.org/abs/2412.13224 · HTML: https://mlsanity.com/paper/2412.13224/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 24 of 24)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Safe Reinforcement Learning in Constrained Markov Decision Processes](https://mlsanity.com/paper/2008.06626/) (2020)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://mlsanity.com/paper/2109.06325/) (2021)
- [Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics](https://mlsanity.com/paper/2107.04989/) (2021)
- [Description of Corner Cases in Automated Driving: Goals and Challenges](https://mlsanity.com/paper/2109.09607/) (2021)
- [Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2208.06721/) (2022)
- [Verification for Machine Learning, Autonomy, and Neural Networks Survey](https://mlsanity.com/paper/1810.01989/) (2018)
- [Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics](https://mlsanity.com/paper/2107.09822/) (2021)
- [Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach](https://mlsanity.com/paper/2304.04066/) (2023)
- [Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings](https://mlsanity.com/paper/2305.16614/) (2023)
- [Safety Verification of Neural Network Controlled Systems](https://mlsanity.com/paper/2011.05174/) (2020)
- [Equipping Black-Box Policies with Model-Based Advice for Stable Nonlinear Control](https://mlsanity.com/paper/2206.01341/) (2022)
- [Simplex-enabled Safe Continual Learning Machine](https://mlsanity.com/paper/2409.05898/) (2024)
- [Synthesis of Admissible Shields](https://mlsanity.com/paper/1904.06938/) (2019)

## Abstract (arXiv)

Real-world accidents in learning-enabled CPS frequently occur in challenging corner cases. During the training of deep reinforcement learning (DRL) policy, the standard setup for training conditions is either fixed at a single initial condition or uniformly sampled from the admissible state space. This setup often overlooks the challenging but safety-critical corner cases. To bridge this gap, this paper proposes a physics-model-guided worst-case sampling strategy for training safe policies that can handle safety-critical cases toward guaranteed safety. Furthermore, we integrate the proposed worst-case sampling strategy into the physics-regulated deep reinforcement learning (Phy-DRL) framework to build a more data-efficient and safe learning algorithm for safety-critical CPS. We validate the proposed training strategy with Phy-DRL through extensive experiments on a simulated cart-pole system, a 2D quadrotor, a simulated and a real quadruped robot, showing remarkably improved sampling efficiency to learn more robust safe policies.