# Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances

arXiv: https://arxiv.org/abs/2409.19746 · HTML: https://mlsanity.com/paper/2409.19746/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 29 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [DeepReach: A Deep Learning Approach to High-Dimensional Reachability](https://mlsanity.com/paper/2011.02082/) (2020)
- [Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control](https://mlsanity.com/paper/2103.02142/) (2021)
- [Robust Deep Reinforcement Learning through Adversarial Loss](https://mlsanity.com/paper/2008.01976/) (2020)
- [Robust Reinforcement Learning using Adversarial Populations](https://mlsanity.com/paper/2008.01825/) (2020)
- [WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contacts](https://mlsanity.com/paper/2406.06005/) (2024)
- [Risk Averse Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1904.00511/) (2019)
- [safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://mlsanity.com/paper/2109.06325/) (2021)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraints](https://mlsanity.com/paper/1410.6445/) (2014)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training](https://mlsanity.com/paper/2202.09514/) (2022)
- [ISAACS: Iterative Soft Adversarial Actor-Critic for Safety](https://mlsanity.com/paper/2212.03228/) (2022)
- [What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery](https://mlsanity.com/paper/2306.15864/) (2023)
- [Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2407.09645/) (2024)
- [OptimizedDP: An Efficient, User-friendly Library For Optimal Control and Dynamic Programming](https://mlsanity.com/paper/2204.05520/) (2022)
- [Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula](https://mlsanity.com/paper/2311.01642/) (2023)
- [Guardians as You Fall: Active Mode Transition for Safe Falling](https://mlsanity.com/paper/2310.04828/) (2023)
- [Exact and Efficient Hamilton-Jacobi-based Guaranteed Safety Analysis via System Decomposition](https://mlsanity.com/paper/1609.05248/) (2016)
- [Multi-Agent Reach-Avoid Games: Two Attackers Versus One Defender and Mixed Integer Programming](https://mlsanity.com/paper/2309.13155/) (2023)
- [Providing Safety Assurances for Systems with Unknown Dynamics](https://mlsanity.com/paper/2403.05771/) (2024)
- [Robust Control Lyapunov-Value Functions for Nonlinear Disturbed Systems](https://mlsanity.com/paper/2403.03455/) (2024)
- [Robust Control using Control Lyapunov Function and Hamilton-Jacobi Reachability](https://mlsanity.com/paper/2404.05625/) (2024)

## Built on by (most cited first, 1 of 1)

- [Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identifications](https://mlsanity.com/paper/2410.20357/) (2024)

## Abstract (arXiv)

Deep Reinforcement Learning (RL) has shown remarkable success in robotics with complex and heterogeneous dynamics. However, its vulnerability to unknown disturbances and adversarial attacks remains a significant challenge. In this paper, we propose a robust policy training framework that integrates model-based control principles with adversarial RL training to improve robustness without the need for external black-box adversaries. Our approach introduces a novel Hamilton-Jacobi reachability-guided disturbance for adversarial RL training, where we use interpretable worst-case or near-worst-case disturbances as adversaries against the robust policy. We evaluated its effectiveness across three distinct tasks: a reach-avoid game in both simulation and real-world settings, and a highly dynamic quadrotor stabilization task in simulation. We validate that our learned critic network is consistent with the ground-truth HJ value function, while the policy network shows comparable performance with other learning-based methods.