# Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

arXiv: https://arxiv.org/abs/2603.12110 · HTML: https://mlsanity.com/paper/2603.12110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 10 of 10)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)
- [Robust Deep Reinforcement Learning Through Adversarial Attacks and Training : A Survey](https://mlsanity.com/paper/2403.00420/) (2024)
- [Learning H-Infinity Locomotion Control](https://mlsanity.com/paper/2404.14405/) (2024)
- [Robust Deep Reinforcement Learning for Quadcopter Control](https://mlsanity.com/paper/2111.03915/) (2021)
- [Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control](https://mlsanity.com/paper/2502.21057/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) policies trained in simulation often degrade once deployed on real systems, where the controller must reject external disturbances that were never encountered in simulation. Robust RL addresses this by exposing the controller to perturbations while it learns, through domain randomization, adversarial minimax formulations, or probabilistic mixtures of protagonist and adversarial behavior. However, an unregulated disturbance mechanism destabilizes training and often collapses nominal performance relative to standard, non-robust methods. We propose cost-to-disturbance ratio adversarial training (CoDRA), a framework that expresses the controller--adversary trade-off as a ratio of accumulated cost to accumulated squared disturbance norm, and optimizes it through a self-normalized actor--critic update. In this algorithm, each value term is scaled by a stop-gradient normalization constant computed from the current batch. This moderates the adversary's incentive without altering the controller's own update, and requires neither an explicit disturbance penalty nor an auxiliary trade-off parameter. We evaluate CoDRA on two MuJoCo pendulum environments under force and mass sweeps. On InvertedDoublePendulum, CoDRA attains the lowest cost at every force level, including a force outside the range seen during training, and in all but one cell of the mass grid, whereas its advantage is less pronounced on the milder InvertedPendulum.