# Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning

arXiv: https://arxiv.org/abs/2208.13663 · HTML: https://mlsanity.com/paper/2208.13663/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 29 Aug 2022.

## Builds on (most cited first, 24 of 24)

- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)
- [Bandits with Knapsacks](https://mlsanity.com/paper/1305.2545/) (2013)
- [Delving into adversarial attacks on deep policies](https://mlsanity.com/paper/1705.06452/) (2017)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks](https://mlsanity.com/paper/1701.04143/) (2017)
- [Adaptive Reward-Poisoning Attacks against Reinforcement Learning](https://mlsanity.com/paper/2003.12613/) (2020)
- [Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning](https://mlsanity.com/paper/2003.12909/) (2020)
- [Data Poisoning against Differentially-Private Learners: Attacks and Defenses](https://mlsanity.com/paper/1903.09860/) (2019)
- [Policy Poisoning in Batch Reinforcement Learning and Control](https://mlsanity.com/paper/1910.05821/) (2019)
- [Stochastic Linear Bandits Robust to Adversarial Attacks](https://mlsanity.com/paper/2007.03285/) (2020)
- [Adversarial Attacks on Stochastic Bandits](https://mlsanity.com/paper/1810.12188/) (2018)
- [Data Poisoning Attacks on Stochastic Bandits](https://mlsanity.com/paper/1905.06494/) (2019)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [Improved Corruption Robust Algorithms for Episodic Reinforcement Learning](https://mlsanity.com/paper/2102.06875/) (2021)
- [Nearly Minimax Optimal Regret for Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation](https://mlsanity.com/paper/2102.07301/) (2021)
- [Saving Stochastic Bandits from Poisoning Attacks via Limited Data Verification](https://mlsanity.com/paper/2102.07711/) (2021)
- [Online learning with feedback graphs and switching costs](https://mlsanity.com/paper/1810.09666/) (2018)
- [Action-Manipulation Attacks Against Stochastic Bandits: Attacks and Defense](https://mlsanity.com/paper/2002.08000/) (2020)
- [Learning-based attacks in Cyber-Physical Systems: Exploration, Detection, and Control Cost trade-offs](https://mlsanity.com/paper/2011.10718/) (2020)
- [Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals](https://mlsanity.com/paper/2002.03827/) (2020)
- [Reinforcement Learning for Unified Allocation and Patrolling in Signaling Games with Uncertainty](https://mlsanity.com/paper/2012.10389/) (2020)

## Built on by (most cited first, 21 of 21)

- [SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents](https://mlsanity.com/paper/2405.20539/) (2024)
- [Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attacks](https://mlsanity.com/paper/2302.13851/) (2023)
- [Optimal Attack and Defense for Reinforcement Learning](https://mlsanity.com/paper/2312.00198/) (2023)
- [Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2307.07670/) (2023)
- [Artwork Protection Against Neural Style Transfer Using Locally Adaptive Adversarial Color Attack](https://mlsanity.com/paper/2401.09673/) (2024)
- [Adversarial Inception Backdoor Attacks against Reinforcement Learning](https://mlsanity.com/paper/2410.13995/) (2024)
- [Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2505.19532/) (2025)
- [Adversarially Robust Decision Transformer](https://mlsanity.com/paper/2407.18414/) (2024)
- [Minimally Modifying a Markov Game to Achieve Any Nash Equilibrium and Value](https://mlsanity.com/paper/2311.00582/) (2023)
- [Adversarial Attacks on Combinatorial Multi-Armed Bandits](https://mlsanity.com/paper/2310.05308/) (2023)
- [Robust Deep Reinforcement Learning against Adversarial Behavior Manipulation](https://mlsanity.com/paper/2406.03862/) (2024)
- [Online Poisoning Attack Against Reinforcement Learning under Black-box Environments](https://mlsanity.com/paper/2412.00797/) (2024)
- [Policy Teaching via Data Poisoning in Learning from Human Preferences](https://mlsanity.com/paper/2503.10228/) (2025)
- [Data Poisoning to Fake a Nash Equilibrium in Markov Games](https://mlsanity.com/paper/2306.08041/) (2023)
- [Inception: Efficiently Computable Misinformation Attacks on Markov Games](https://mlsanity.com/paper/2406.17114/) (2024)
- [Efficient Preference Poisoning Attack on Offline RLHF](https://mlsanity.com/paper/2605.02495/) (2026)
- [Optimally Installing Strict Equilibria](https://mlsanity.com/paper/2503.03676/) (2025)
- [Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks](https://mlsanity.com/paper/2506.06891/) (2025)
- [Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning](https://mlsanity.com/paper/2506.11172/) (2025)
- [Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback](https://mlsanity.com/paper/2603.28281/) (2026)
- [When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPs](https://mlsanity.com/paper/2604.10062/) (2026)

## Abstract (arXiv)

To understand the security threats to reinforcement learning (RL) algorithms, this paper studies poisoning attacks to manipulate any order-optimal learning algorithm towards a targeted policy in episodic RL and examines the potential damage of two natural types of poisoning attacks, i.e., the manipulation of reward and action. We discover that the effect of attacks crucially depend on whether the rewards are bounded or unbounded. In bounded reward settings, we show that only reward manipulation or only action manipulation cannot guarantee a successful attack. However, by combining reward and action manipulation, the adversary can manipulate any order-optimal learning algorithm to follow any targeted policy with tildeTheta(√T) total attack cost, which is order-optimal, without any knowledge of the underlying MDP. In contrast, in unbounded reward settings, we show that reward manipulation attacks are sufficient for an adversary to successfully manipulate any order-optimal learning algorithm to follow any targeted policy using tildeO(√T) amount of contamination. Our results reveal useful insights about what can or cannot be achieved by poisoning attacks, and are set to spur more works on the design of robust RL algorithms.