# ISAACS: Iterative Soft Adversarial Actor-Critic for Safety

arXiv: https://arxiv.org/abs/2212.03228 · HTML: https://mlsanity.com/paper/2212.03228/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 22 of 22)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances](https://mlsanity.com/paper/1709.07523/) (2017)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [DeepReach: A Deep Learning Approach to High-Dimensional Reachability](https://mlsanity.com/paper/2011.02082/) (2020)
- [Robust Reinforcement Learning using Adversarial Populations](https://mlsanity.com/paper/2008.01825/) (2020)
- [Who Leads and Who Follows in Strategic Classification?](https://mlsanity.com/paper/2106.12529/) (2021)
- [Robust Control Barrier-Value Functions for Safety-Critical Control](https://mlsanity.com/paper/2104.02808/) (2021)
- [Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning](https://mlsanity.com/paper/2112.12288/) (2021)
- [Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraints](https://mlsanity.com/paper/1410.6445/) (2014)
- [Linear model predictive safety certification for learning-based control](https://mlsanity.com/paper/1803.08552/) (2018)
- [Robustness of Control Barrier Functions for Safety Critical Control](https://mlsanity.com/paper/1612.01554/) (2016)
- [Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees](https://mlsanity.com/paper/2201.08355/) (2022)
- [Safety-Critical Control using Optimal-decay Control Barrier Function with Guaranteed Point-wise Feasibility](https://mlsanity.com/paper/2103.12375/) (2021)
- [An Iterative Abstraction Algorithm for Reactive Correct-by-Construction Controller Synthesis](https://mlsanity.com/paper/1509.04125/) (2015)

## Built on by (most cited first, 28 of 28)

- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination](https://mlsanity.com/paper/2405.00846/) (2024)
- [Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2407.09645/) (2024)
- [SAFE-GIL: SAFEty Guided Imitation Learning for Robotic Systems](https://mlsanity.com/paper/2404.05249/) (2024)
- [Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanes](https://mlsanity.com/paper/2402.05279/) (2024)
- [Learning Vision-based Pursuit-Evasion Robot Policies](https://mlsanity.com/paper/2308.16185/) (2023)
- [Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy](https://mlsanity.com/paper/2309.01267/) (2023)
- [Gradient Shaping for Multi-Constraint Safe Reinforcement Learning](https://mlsanity.com/paper/2312.15127/) (2023)
- [Preemptive Detection and Steering of LLM Misalignment via Latent Reachability](https://mlsanity.com/paper/2509.21528/) (2025)
- [How to Train Your Latent Control Barrier Function: Smooth Safety Filtering Under Hard-to-Model Constraints](https://mlsanity.com/paper/2511.18606/) (2025)
- [Human-AI Safety: A Descendant of Generative AI and Control Systems Safety](https://mlsanity.com/paper/2405.09794/) (2024)
- [MAGICS: Adversarial RL with Minimax Actors Guided by Implicit Critic Stackelberg for Convergent Neural Synthesis of Robot Safety](https://mlsanity.com/paper/2409.13867/) (2024)
- [Synthesis and Deployment of Maximal Robust Control Barrier Functions through Adversarial Reinforcement Learning](https://mlsanity.com/paper/2604.13192/) (2026)
- [Provably-Safe Neural Network Training Using Hybrid Zonotope Reachability Analysis](https://mlsanity.com/paper/2501.13023/) (2025)
- [Your Learned Constraint is Secretly a Backward Reachable Tube](https://mlsanity.com/paper/2501.15618/) (2025)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)
- [Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances](https://mlsanity.com/paper/2409.19746/) (2024)
- [Deep Reinforcement Learning for Reach-Avoid-Stay Problems](https://mlsanity.com/paper/2410.02898/) (2024)
- [From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails](https://mlsanity.com/paper/2510.13727/) (2025)
- [Permissive Safety Through Trusted Inference: Verifiable Belief-Space Neural Safety Filters for Assured Interactive Robotics](https://mlsanity.com/paper/2606.02562/) (2026)
- [Robots that Suggest Safe Alternatives](https://mlsanity.com/paper/2409.09883/) (2024)
- [Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning](https://mlsanity.com/paper/2602.15817/) (2026)
- [TAIL-Safe: Task-Agnostic Safety Monitoring for Imitation Learning Policies](https://mlsanity.com/paper/2605.01195/) (2026)
- [ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids](https://mlsanity.com/paper/2609.15988/) (2026)
- [Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning](https://mlsanity.com/paper/2609.27312/) (2026)
- [LEAP-CBF: A Safety Filter for Uncertain Systems with Least-Effort Adversarial Potentials](https://mlsanity.com/paper/2609.28364/) (2026)

## Abstract (arXiv)

The deployment of robots in uncontrolled environments requires them to operate robustly under previously unseen scenarios, like irregular terrain and wind conditions. Unfortunately, while rigorous safety frameworks from robust optimal control theory scale poorly to high-dimensional nonlinear dynamics, control policies computed by more tractable "deep" methods lack guarantees and tend to exhibit little robustness to uncertain operating conditions. This work introduces a novel approach enabling scalable synthesis of robust safety-preserving controllers for robotic systems with general nonlinear dynamics subject to bounded modeling error by combining game-theoretic safety analysis with adversarial reinforcement learning in simulation. Following a soft actor-critic scheme, a safety-seeking fallback policy is co-trained with an adversarial "disturbance" agent that aims to invoke the worst-case realization of model error and training-to-deployment discrepancy allowed by the designer's uncertainty. While the learned control policy does not intrinsically guarantee safety, it is used to construct a real-time safety filter (or shield) with robust safety guarantees based on forward reachability rollouts. This shield can be used in conjunction with a safety-agnostic control policy, precluding any task-driven actions that could result in loss of safety. We evaluate our learning-based safety approach in a 5D race car simulator, compare the learned safety policy to the numerically obtained optimal solution, and empirically validate the robust safety guarantee of our proposed safety shield against worst-case model discrepancy.