# Risk Averse Robust Adversarial Reinforcement Learning

arXiv: https://arxiv.org/abs/1904.00511 · HTML: https://mlsanity.com/paper/1904.00511/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2019.

## Built on by (most cited first, 30 of 57)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.12581/) (2022)
- [Risk-Averse Offline Reinforcement Learning](https://mlsanity.com/paper/2102.05371/) (2021)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Risk-Averse Bayes-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2102.05762/) (2021)
- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [Adversarial Training is Not Ready for Robot Learning](https://mlsanity.com/paper/2103.08187/) (2021)
- [Risk-Sensitive Reinforcement Learning with Exponential Criteria](https://mlsanity.com/paper/2212.09010/) (2022)
- [Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers](https://mlsanity.com/paper/2305.05909/) (2023)
- [Robust Reinforcement Learning Under Minimax Regret for Green Security](https://mlsanity.com/paper/2106.08413/) (2021)
- [RAMP-Net: A Robust Adaptive MPC for Quadrotors via Physics-informed Neural Network](https://mlsanity.com/paper/2209.09025/) (2022)
- [Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments](https://mlsanity.com/paper/2504.19139/) (2025)
- [Robust Reinforcement Learning via Genetic Curriculum](https://mlsanity.com/paper/2202.08393/) (2022)
- [Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula](https://mlsanity.com/paper/2311.01642/) (2023)
- [CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observations](https://mlsanity.com/paper/2311.03485/) (2023)
- [Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernel](https://mlsanity.com/paper/2306.05859/) (2023)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generation](https://mlsanity.com/paper/2305.05116/) (2023)
- [Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation](https://mlsanity.com/paper/2504.20887/) (2025)
- [Instance-Aware Predictive Navigation in Multi-Agent Environments](https://mlsanity.com/paper/2101.05893/) (2021)
- [Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory](https://mlsanity.com/paper/2308.13011/) (2023)
- [Safe and Robust Reinforcement Learning: Principles and Practice](https://mlsanity.com/paper/2403.18539/) (2024)
- [ARC: Adversarially Robust Control Policies for Autonomous Vehicles](https://mlsanity.com/paper/2107.04487/) (2021)
- [Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets](https://mlsanity.com/paper/2107.08083/) (2021)
- [Robust Dynamic Bus Control: A Distributional Multi-agent Reinforcement Learning Approach](https://mlsanity.com/paper/2111.01946/) (2021)
- [Reward-Free Attacks in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2112.00940/) (2021)
- [Robust Multi-agent Communication via Multi-view Message Certification](https://mlsanity.com/paper/2305.13936/) (2023)
- [DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty](https://mlsanity.com/paper/2506.12622/) (2025)
- [ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving](https://mlsanity.com/paper/2603.15221/) (2026)
- [Robust Reinforcement Learning under model misspecification](https://mlsanity.com/paper/2103.15370/) (2021)

## Abstract (arXiv)

Deep reinforcement learning has recently made significant progress in solving computer games and robotic control tasks. A known problem, though, is that policies overfit to the training environment and may not avoid rare, catastrophic events such as automotive accidents. A classical technique for improving the robustness of reinforcement learning algorithms is to train on a set of randomized environments, but this approach only guards against common situations. Recently, robust adversarial reinforcement learning (RARL) was developed, which allows efficient applications of random and systematic perturbations by a trained adversary. A limitation of RARL is that only the expected control objective is optimized; there is no explicit modeling or optimization of risk. Thus the agents do not consider the probability of catastrophic events (i.e., those inducing abnormally large negative reward), except through their effect on the expected objective. In this paper we introduce risk-averse robust adversarial reinforcement learning (RARARL), using a risk-averse protagonist and a risk-seeking adversary. We test our approach on a self-driving vehicle controller. We use an ensemble of policy networks to model risk as the variance of value functions. We show through experiments that a risk-averse agent is better equipped to handle a risk-seeking adversary, and experiences substantially fewer crashes compared to agents trained without an adversary.