# Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers

arXiv: https://arxiv.org/abs/2106.07798 · HTML: https://mlsanity.com/paper/2106.07798/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [Blind Backdoors in Deep Learning Models](https://mlsanity.com/paper/2005.03823/) (2020)
- [NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations](https://mlsanity.com/paper/1911.07399/) (2019)
- [The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models](https://mlsanity.com/paper/2003.07233/) (2020)
- [Natural Backdoor Attack on Text Data](https://mlsanity.com/paper/2006.16176/) (2020)
- [Invisible Backdoor Attacks on Deep Neural Networks via Steganography and Regularization](https://mlsanity.com/paper/1909.02742/) (2019)

## Built on by (most cited first, 20 of 20)

- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets](https://mlsanity.com/paper/2210.04688/) (2022)
- [A Temporal-Pattern Backdoor Attack to Deep Reinforcement Learning](https://mlsanity.com/paper/2205.02589/) (2022)
- [Adversarial Cheap Talk](https://mlsanity.com/paper/2211.11030/) (2022)
- [BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systems](https://mlsanity.com/paper/2501.01593/) (2025)
- [A Spatiotemporal Stealthy Backdoor Attack against Cooperative Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2409.07775/) (2024)
- [Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2505.19532/) (2025)
- [Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space](https://mlsanity.com/paper/2407.15168/) (2024)
- [Backdoor Attacks on Multiagent Collaborative Systems](https://mlsanity.com/paper/2211.11455/) (2022)
- [Don't Watch Me: A Spatio-Temporal Trojan Attack on Deep-Reinforcement-Learning-Augment Autonomous Driving](https://mlsanity.com/paper/2211.14440/) (2022)
- [The last Dance : Robust backdoor attack via diffusion models and bayesian approach](https://mlsanity.com/paper/2402.05967/) (2024)
- [TooBadRL: Trigger Optimization to Boost Effectiveness of Backdoor Attacks on Deep Reinforcement Learning](https://mlsanity.com/paper/2506.09562/) (2025)
- [TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models](https://mlsanity.com/paper/2506.12815/) (2025)
- [Security of Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2212.06123/) (2022)
- [Online Poisoning Attack Against Reinforcement Learning under Black-box Environments](https://mlsanity.com/paper/2412.00797/) (2024)
- [Backdoors in DRL: Four Environments Focusing on In-distribution Triggers](https://mlsanity.com/paper/2505.17248/) (2025)
- [Investigating the Treacherous Turn in Deep Reinforcement Learning](https://mlsanity.com/paper/2504.08943/) (2025)
- [Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04883/) (2025)
- [Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity](https://mlsanity.com/paper/2606.00414/) (2026)

## Abstract (arXiv)

In this paper, we propose a new data poisoning attack and apply it to deep reinforcement learning agents. Our attack centers on what we call in-distribution triggers, which are triggers native to the data distributions the model will be trained on and deployed in. We outline a simple procedure for embedding these, and other, triggers in deep reinforcement learning agents following a multi-task learning paradigm, and demonstrate in three common reinforcement learning environments. We believe that this work has important implications for the security of deep learning models.