# Reward-Preserving Attacks For Robust Reinforcement Learning

arXiv: https://arxiv.org/abs/2601.07118 · HTML: https://mlsanity.com/paper/2601.07118/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 17 of 17)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Wasserstein Robust Reinforcement Learning](https://mlsanity.com/paper/1907.13196/) (2019)
- [Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning](https://mlsanity.com/paper/1903.03642/) (2019)
- [Distributionally Robust Policy Learning via Adversarial Environment Generation](https://mlsanity.com/paper/2107.06353/) (2021)
- [Robust Deep Reinforcement Learning Through Adversarial Attacks and Training : A Survey](https://mlsanity.com/paper/2403.00420/) (2024)
- [Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification](https://mlsanity.com/paper/2211.03413/) (2022)
- [Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space](https://mlsanity.com/paper/2405.11982/) (2024)
- [Time-Constrained Robust MDPs](https://mlsanity.com/paper/2406.08395/) (2024)

## Abstract (arXiv)

Adversarial training in reinforcement learning (RL) is challenging because perturbations cascade through trajectories and compound over time, making fixed-strength attacks either overly destructive or too conservative. We propose reward-preserving attacks, which adapt adversarial strength so that an α fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, perturbation magnitudes η are selected dynamically, using a learned critic Q((s,a),η) that estimates the expected return of α-reward-preserving rollouts. For intermediate values of α, this adaptive training yields policies that are robust across a wide range of perturbation magnitudes while preserving nominal performance, outperforming fixed-radius and uniformly sampled-radius adversarial training.