# Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees

arXiv: https://arxiv.org/abs/2201.08355 · HTML: https://mlsanity.com/paper/2201.08355/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 74 references that are in the map.
- First posted on arXiv 20 Jan 2022.

## Builds on (most cited first, 30 of 51)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances](https://mlsanity.com/paper/1709.07523/) (2017)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [A Primer on PAC-Bayesian Learning](https://mlsanity.com/paper/1901.05353/) (2019)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Tighter risk certificates for neural networks](https://mlsanity.com/paper/2007.12911/) (2020)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators](https://mlsanity.com/paper/1906.01728/) (2019)

## Built on by (most cited first, 27 of 27)

- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [ISAACS: Iterative Soft Adversarial Actor-Critic for Safety](https://mlsanity.com/paper/2212.03228/) (2022)
- [NavRL: Learning Safe Flight in Dynamic Environments](https://mlsanity.com/paper/2409.15634/) (2024)
- [AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer](https://mlsanity.com/paper/2302.04903/) (2023)
- [Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination](https://mlsanity.com/paper/2405.00846/) (2024)
- [Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2407.09645/) (2024)
- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [Gradient Shaping for Multi-Constraint Safe Reinforcement Learning](https://mlsanity.com/paper/2312.15127/) (2023)
- [One Filter to Deploy Them All: Robust Safety for Quadrupedal Navigation in Unknown Environments](https://mlsanity.com/paper/2412.09989/) (2024)
- [How to Train Your Latent Control Barrier Function: Smooth Safety Filtering Under Hard-to-Model Constraints](https://mlsanity.com/paper/2511.18606/) (2025)
- [Learning Control Barrier Functions and their application in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2404.16879/) (2024)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [AnySafe: Adapting Latent Safety Filters at Runtime via Safety Constraint Parameterization in the Latent Space](https://mlsanity.com/paper/2509.19555/) (2025)
- [AnyBipe: An End-to-End Framework for Training and Deploying Bipedal Robots Guided by Large Language Models](https://mlsanity.com/paper/2409.08904/) (2024)
- [A Multiplicative Value Function for Safe and Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.04118/) (2023)
- [Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2405.11669/) (2024)
- [A Provable Approach for End-to-End Safe Reinforcement Learning](https://mlsanity.com/paper/2505.21852/) (2025)
- [Deep Reinforcement Learning for Reach-Avoid-Stay Problems](https://mlsanity.com/paper/2410.02898/) (2024)
- [Neural Fidelity Calibration for Informative Sim-to-Real Adaptation](https://mlsanity.com/paper/2504.08604/) (2025)
- [Guided by Guardrails: Control Barrier Functions as Safety Instructors for Robotic Learning](https://mlsanity.com/paper/2505.18858/) (2025)
- [Deep Actor-Critics with Tight Risk Certificates](https://mlsanity.com/paper/2505.19682/) (2025)
- [Reversible Simplex Supervision with Post-Action Debt Accounting for Goal-Reaching RL](https://mlsanity.com/paper/2601.19499/) (2026)
- [Safety Guarantees in Zero-Shot Reinforcement Learning for Cascade Dynamical Systems](https://mlsanity.com/paper/2604.10429/) (2026)
- [Shield-Loco: Shielding Locomotion Policies with Predictive Safety Filtering](https://mlsanity.com/paper/2606.07193/) (2026)

## Abstract (arXiv)

Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to bridge the reality gap with a probabilistically guaranteed safety-aware policy distribution. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the Safety Bellman Equation based on Hamilton-Jacobi (HJ) reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. Additionally, inheriting from the HJ reachability analysis, the bound accounts for the expectation over the worst-case safety in each environment. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments with varying degrees of photorealism. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.