# Sequential Attacks on Agents for Long-Term Adversarial Goals

arXiv: https://arxiv.org/abs/1805.12487 · HTML: https://mlsanity.com/paper/1805.12487/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2018.

## Built on by (most cited first, 16 of 16)

- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Defense Against Reward Poisoning Attacks in Reinforcement Learning](https://mlsanity.com/paper/2102.05776/) (2021)
- [Optimal Attack and Defense for Reinforcement Learning](https://mlsanity.com/paper/2312.00198/) (2023)
- [Resilient Machine Learning for Networked Cyber Physical Systems: A Survey for Machine Learning Security to Securing Machine Learning for CPS](https://mlsanity.com/paper/2102.07244/) (2021)
- [Real-time Adversarial Perturbations against Deep Reinforcement Learning Policies: Attacks and Defenses](https://mlsanity.com/paper/2106.08746/) (2021)
- [Sparse Adversarial Attack in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2205.09362/) (2022)
- [Deep-Attack over the Deep Reinforcement Learning](https://mlsanity.com/paper/2205.00807/) (2022)
- [Emerging Threats in Deep Learning-Based Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2210.11237/) (2022)
- [SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2301.04299/) (2023)
- [Enhancing the Robustness of QMIX against State-adversarial Attacks](https://mlsanity.com/paper/2307.00907/) (2023)
- [Advancing Robustness in Deep Reinforcement Learning with an Ensemble Defense Approach](https://mlsanity.com/paper/2507.17070/) (2025)
- [Security of Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2212.06123/) (2022)
- [Robust Deep Reinforcement Learning against Adversarial Behavior Manipulation](https://mlsanity.com/paper/2406.03862/) (2024)
- [A Transferable and Automatic Tuning of Deep Reinforcement Learning for Cost Effective Phishing Detection](https://mlsanity.com/paper/2209.09033/) (2022)
- [Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learning](https://mlsanity.com/paper/2510.12939/) (2025)
- [Adversarially Robust PAC Learning with Optimal VC Rates](https://mlsanity.com/paper/2609.24260/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has advanced greatly in the past few years with the employment of effective deep neural networks (DNNs) on the policy networks. With the great effectiveness came serious vulnerability issues with DNNs that small adversarial perturbations on the input can change the output of the network. Several works have pointed out that learned agents with a DNN policy network can be manipulated against achieving the original task through a sequence of small perturbations on the input states. In this paper, we demonstrate furthermore that it is also possible to impose an arbitrary adversarial reward on the victim policy network through a sequence of attacks. Our method involves the latest adversarial attack technique, Adversarial Transformer Network (ATN), that learns to generate the attack and is easy to integrate into the policy network. As a result of our attack, the victim agent is misguided to optimise for the adversarial reward over time. Our results expose serious security threats for RL applications in safety-critical systems including drones, medical analysis, and self-driving cars.