Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty
Not on the map. Why?
Abstract (arXiv)
Safety remains an open problem in reinforcement learning (RL), especially during training. While safety filters are promising to address safe exploration, they are generally poorly suited for high-dimensional systems with unknown dynamics. We propose Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a novel algorithm that learns both a scalable safety filter and a control policy using a learned uncertainty-aware dynamics model, while requiring minimal domain knowledge. The filter avoids failures and high uncertainty regions. Thus, better models expand the set of safe and certain states, reducing filter conservatism. We present the effectiveness of Dyna-SAuR on goal-reaching CartPole as well as MuJoCo Walker, reducing failures compared to state-of-the-art methods by 2 orders of magnitude.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 28 Apr 2026.
Builds on
37 of 51 references.
References · 37
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2017Constrained Policy Optimizationcited by 928
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2018Reward Constrained Policy Optimizationcited by 397
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
- 2019Control Barrier Functions: Theory and Applicationscited by 332
- 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
- 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
- 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
25 more
- 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
- 2020Conservative Safety Critics for Explorationcited by 107
- 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
- 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 61
- 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
- 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
- 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
- 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
- 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 44
- 2023How to Train Your Neural Control Barrier Function: Learning Safety Filters for Complex Input-Constrained Systemscited by 40
- 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
- 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
- 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
- 2020Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Controlcited by 28
- 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
- 2021Safe Control with Neural Network Dynamic Modelscited by 21
- 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
- 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
- 2024Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agentscited by 11
- 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
- 2024Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanescited by 8
- 2021Safe Value Functionscited by 7
- 2025Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?cited by 5
- 2019Computing Probabilistic Controlled Invariant Setscited by 4
- 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularizationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.