Data to 7 Oct 2026

Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning

arXiv 2003.12909 ↗First posted 28 Mar 2020cs.LG cs.AI cs.CRcited by 81

Not on the map. Why?

Abstract (arXiv)

We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes average reward in undiscounted infinite-horizon problem settings. The attacker can manipulate the rewards or the transition dynamics in the learning environment at training-time and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an optimal stealthy attack for different measures of attack cost. We provide sufficient technical conditions under which the attack is feasible and provide lower/upper bounds on the attack cost. We instantiate our attacks in two settings: (i) an offline setting where the agent is doing planning in the poisoned environment, and (ii) an online setting where the agent is learning a policy using a regret-minimization framework with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 14; 2023: 16; 2024: 16; 2025: 13; 2026: 6.

81 papers.

Sort:

By area:

2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 93
2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 54
2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
2021Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learningcited by 29
2023BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learningcited by 29
2021Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environmentscited by 26
2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attackscited by 25
2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 20

38 more

2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
2024SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agentscited by 19
2021Reinforcement Learning for Feedback-Enabled Cyber Resiliencecited by 18
2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 16
2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
2024Nonparametric Teaching of Implicit Neural Representationscited by 13
2022Admissible Policy Teaching through Reward Designcited by 12
2023Principal-Agent Reward Shaping in MDPscited by 12
2021Fair Clustering Using Antidote Datacited by 10
2021When Are Linear Stochastic Bandits Attackable?cited by 10
2021Iterative Teaching by Label Synthesiscited by 10
2022Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learningcited by 9
2023Nonparametric Iterative Machine Teachingcited by 9
2023Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learningcited by 9
2021Game Redesign in No-regret Game Playingcited by 8
2022Iterative Teaching by Data Hallucinationcited by 8
2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
2024Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policiescited by 8
2022Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenariocited by 7
2022New Challenges in Reinforcement Learning: A Survey of Security and Privacycited by 7
2023Nonparametric Teaching for Multiple Learnerscited by 7
2023Local Environment Poisoning Attacks on Federated Reinforcement Learningcited by 6
2022Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checkingcited by 5
2023Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learningcited by 5
2025Nonparametric Teaching for Graph Property Learnerscited by 5
2025Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learningcited by 5
2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performancecited by 4
2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
2024Corruption-Robust Offline Two-Player Zero-Sum Markov Gamescited by 4
2024Data Poisoning Attacks on Off-Policy Policy Evaluation Methodscited by 4
2024Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Spacecited by 4
2021Efficient Action Poisoning Attacks on Linear Contextual Banditscited by 3
2023Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theoremscited by 3
2024Informativeness of Reward Functions in Reinforcement Learningcited by 3
2022Imitating Opponent to Win: Adversarial Policy Imitation Learning in Two-player Competitive Gamescited by 2
2023Adversarial Attacks on Combinatorial Multi-Armed Banditscited by 2
2023Minimally Modifying a Markov Game to Achieve Any Nash Equilibrium and Valuecited by 2

50 most cited of 81 All 81 (JSON)