# Improving Robustness of Deep Reinforcement Learning Agents: Environment Attack based on the Critic Network

arXiv: https://arxiv.org/abs/2104.03154 · HTML: https://mlsanity.com/paper/2104.03154/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks](https://mlsanity.com/paper/1701.04143/) (2017)
- [Reinforcement Learning with Perturbed Rewards](https://mlsanity.com/paper/1810.01032/) (2018)
- [Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals](https://mlsanity.com/paper/1906.10571/) (2019)
- [Online Robustness Training for Deep Reinforcement Learning](https://mlsanity.com/paper/1911.00887/) (2019)
- [Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning](https://mlsanity.com/paper/1903.03642/) (2019)
- [Stochastic sparse adversarial attacks](https://mlsanity.com/paper/2011.12423/) (2020)

## Built on by (most cited first, 3 of 3)

- [Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors](https://mlsanity.com/paper/2408.03200/) (2024)
- [Where2Start: Leveraging initial States for Robust and Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2311.15089/) (2023)
- [Towards Robust Deep Reinforcement Learning against Environmental State Perturbation](https://mlsanity.com/paper/2506.08961/) (2025)

## Abstract (arXiv)

To improve policy robustness of deep reinforcement learning agents, a line of recent works focus on producing disturbances of the environment. Existing approaches of the literature to generate meaningful disturbances of the environment are adversarial reinforcement learning methods. These methods set the problem as a two-player game between the protagonist agent, which learns to perform a task in an environment, and the adversary agent, which learns to disturb the protagonist via modifications of the considered environment. Both protagonist and adversary are trained with deep reinforcement learning algorithms. Alternatively, we propose in this paper to build on gradient-based adversarial attacks, usually used for classification tasks for instance, that we apply on the critic network of the protagonist to identify efficient disturbances of the environment. Rather than learning an attacker policy, which usually reveals as very complex and unstable, we leverage the knowledge of the critic network of the protagonist, to dynamically complexify the task at each step of the learning process. We show that our method, while being faster and lighter, leads to significantly better improvements in policy robustness than existing methods of the literature.