# Safe Reinforcement Learning with Dual Robustness

arXiv: https://arxiv.org/abs/2309.06835 · HTML: https://mlsanity.com/paper/2309.06835/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Reinforcement Learning with Perturbed Rewards](https://mlsanity.com/paper/1810.01032/) (2018)
- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [On the Robustness of Safe Reinforcement Learning under Observational Perturbations](https://mlsanity.com/paper/2205.14691/) (2022)
- [Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligence](https://mlsanity.com/paper/2103.10290/) (2021)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions](https://mlsanity.com/paper/1908.01883/) (2019)
- [RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states](https://mlsanity.com/paper/2205.07229/) (2022)
- [Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotion](https://mlsanity.com/paper/2107.01908/) (2021)
- [Convex computation of the maximum controlled invariant set for polynomial control systems](https://mlsanity.com/paper/1303.6469/) (2013)

## Built on by (most cited first, 10 of 10)

- [RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models](https://mlsanity.com/paper/2511.01331/) (2025)
- [Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium](https://mlsanity.com/paper/2411.15036/) (2024)
- [A Survey of Reinforcement Learning for Optimization in Automation](https://mlsanity.com/paper/2502.09417/) (2025)
- [MULE: Multi-terrain and Unknown Load Adaptation for Effective Quadrupedal Locomotion](https://mlsanity.com/paper/2505.00488/) (2025)
- [Provably Safe, Yet Scalable Reinforcement Learning](https://mlsanity.com/paper/2606.14536/) (2026)
- [Constrained Optimal Fuel Consumption of HEVs under Observational Noise](https://mlsanity.com/paper/2410.20913/) (2024)
- [A Dynamical Systems Framework for Reinforcement Learning Safety and Robustness Verification](https://mlsanity.com/paper/2508.15588/) (2025)
- [Game-Theoretic Risk-Shaped Reinforcement Learning for Safe Autonomous Driving](https://mlsanity.com/paper/2510.10960/) (2025)
- [Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty](https://mlsanity.com/paper/2602.20729/) (2026)
- [Safe Meta-Reinforcement Learning via Information Space Reachability](https://mlsanity.com/paper/2609.15915/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or compromise safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. Optimality is only valid inside a feasible region, while identification of maximal feasible region must rely on learning the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including problem formulation, iteration scheme, convergence analysis and practical algorithm design. This unification is built upon constrained two-player zero-sum Markov games. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. The convergence of this iteration scheme is proved. Furthermore, we design a deep RL algorithm for practical implementation, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines significantly.