# Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions

arXiv: https://arxiv.org/abs/2306.05873 · HTML: https://mlsanity.com/paper/2306.05873/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)
- [EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples](https://mlsanity.com/paper/1709.04114/) (2017)
- [Delving into adversarial attacks on deep policies](https://mlsanity.com/paper/1705.06452/) (2017)
- [The Odds are Odd: A Statistical Test for Detecting Adversarial Examples](https://mlsanity.com/paper/1902.04818/) (2019)
- [Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning](https://mlsanity.com/paper/2005.07099/) (2020)
- [A New Defense Against Adversarial Images: Turning a Weakness into a Strength](https://mlsanity.com/paper/1910.07629/) (2019)
- [An Alternative Surrogate Loss for PGD-based Adversarial Testing](https://mlsanity.com/paper/1910.09338/) (2019)
- [On the Hardness of Robust Classification](https://mlsanity.com/paper/1909.05822/) (2019)
- [Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness](https://mlsanity.com/paper/2301.07487/) (2023)
- [Investigating Vulnerabilities of Deep Neural Policies](https://mlsanity.com/paper/2108.13093/) (2021)
- [Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness](https://mlsanity.com/paper/1905.12202/) (2019)
- [Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs](https://mlsanity.com/paper/2112.09025/) (2021)

## Built on by (most cited first, 7 of 7)

- [A Survey Analyzing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/2401.02349/) (2024)
- [Understanding and Diagnosing Deep Reinforcement Learning](https://mlsanity.com/paper/2406.16979/) (2024)
- [Benchmarking Deep Reinforcement Learning for Navigation in Denied Sensor Environments](https://mlsanity.com/paper/2410.14616/) (2024)
- [Safety, Security, and Cognitive Risks in World Models](https://mlsanity.com/paper/2604.01346/) (2026)
- [Diffusion Guided Adversarial State Perturbations in Reinforcement Learning](https://mlsanity.com/paper/2511.07701/) (2025)
- [Interpretable Failure Analysis in Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2602.08104/) (2026)
- [Boosting Transferable Adversarial Attacks against Deep Reinforcement Learning](https://mlsanity.com/paper/2610.06083/) (2026)

## Abstract (arXiv)

Learning in MDPs with highly complex state representations is currently possible due to multiple advancements in reinforcement learning algorithm design. However, this incline in complexity, and furthermore the increase in the dimensions of the observation came at the cost of volatility that can be taken advantage of via adversarial attacks (i.e. moving along worst-case directions in the observation space). To solve this policy instability problem we propose a novel method to detect the presence of these non-robust directions via local quadratic approximation of the deep neural policy loss. Our method provides a theoretical basis for the fundamental cut-off between safe observations and adversarial observations. Furthermore, our technique is computationally efficient, and does not depend on the methods used to produce the worst-case directions. We conduct extensive experiments in the Arcade Learning Environment with several different adversarial attack techniques. Most significantly, we demonstrate the effectiveness of our approach even in the setting where non-robust directions are explicitly optimized to circumvent our proposed method.