mlsanityData to 7 Oct 2026

Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments

arXiv 2102.08492 ↗First posted 16 Feb 2021cs.LG cs.AI cs.CRcited by 26

Not on the map. Why?

Abstract (arXiv)

We study black-box reward poisoning attacks against reinforcement learning (RL), in which an adversary aims to manipulate the rewards to mislead a sequence of RL agents with unknown algorithms to learn a nefarious policy in an environment unknown to the adversary a priori. That is, our attack makes minimum assumptions on the prior knowledge of the adversary: it has no initial knowledge of the environment or the learner, and neither does it observe the learner's internal mechanism except for its performed actions. We design a novel black-box attack, U2, that can provably achieve a near-matching performance to the state-of-the-art white-box attack, demonstrating the feasibility of reward poisoning even in the most challenging black-box setting.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 40 of its 63 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

40 of 63 references.

References · 40

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2018Is Q-learning Provably Efficient?cited by 485
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  6. 2017Adversarial Attacks on Neural Network Policiescited by 314
  7. 2016Cooperative Inverse Reinforcement Learningcited by 298
  8. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  9. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  10. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  11. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  12. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
28 more
  1. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  2. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  3. 2017Delving into adversarial attacks on deep policiescited by 104
  4. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  5. 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
  6. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  7. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  8. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  9. 2018An Overview of Machine Teachingcited by 63
  10. 2018Adversarial Attacks on Stochastic Banditscited by 56
  11. 2019Data Poisoning Attacks on Stochastic Banditscited by 52
  12. 2019Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signalscited by 50
  13. 2016On Lower Bounds for Regret in Reinforcement Learningcited by 39
  14. 2020Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamicscited by 38
  15. 2020Task-agnostic Exploration in Reinforcement Learningcited by 37
  16. 2016Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Accesscited by 36
  17. 2014Near-Optimally Teaching the Crowd to Classifycited by 32
  18. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applicationscited by 28
  19. 2019Interactive Teaching Algorithms for Inverse Reinforcement Learningcited by 24
  20. 2020Policy Teaching in Reinforcement Learning via Environment Poisoning Attackscited by 24
  21. 2018Best Arm Identification for Contaminated Banditscited by 23
  22. 2020Understanding the Power and Limitations of Teaching with Imperfect Knowledgecited by 23
  23. 2018Data Poisoning Attacks in Contextual Banditscited by 22
  24. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrationscited by 19
  25. 2019Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraintscited by 18
  26. 2018Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learnerscited by 16
  27. 2019Preference-Based Batch and Sequential Teaching: Towards a Unified View of Modelscited by 12
  28. 2018Machine Teaching of Active Sequential Learnerscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 6; 2024: 3; 2025: 4; 2026: 3.

26 papers.

Sort:

By area:

  1. 2021Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learningcited by 29
  2. 2023BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learningcited by 29
  3. 2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
  4. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  5. 2022A Search-Based Testing Approach for Deep Reinforcement Learning Agentscited by 12
  6. 2021When Are Linear Stochastic Bandits Attackable?cited by 10
  7. 2022Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learningcited by 9
  8. 2023Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learningcited by 9
  9. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  10. 2025UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learningcited by 5
  11. 2021Efficient Action Poisoning Attacks on Linear Contextual Banditscited by 3
  12. 2022A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learningcited by 2
14 more
  1. 2022Imitating Opponent to Win: Adversarial Policy Imitation Learning in Two-player Competitive Gamescited by 2
  2. 2023Adversarial Attacks on Combinatorial Multi-Armed Banditscited by 2
  3. 2023Minimally Modifying a Markov Game to Achieve Any Nash Equilibrium and Valuecited by 2
  4. 2024Online Poisoning Attack Against Reinforcement Learning under Black-box Environmentscited by 2
  5. 2022Security of Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 1
  6. 2023Data Poisoning to Fake a Nash Equilibrium in Markov Gamescited by 1
  7. 2023Optimal Cost Constrained Adversarial Attacks For Multiple Agent Systemscited by 1
  8. 2024Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graphcited by 1
  9. 2026Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learningcited by 1
  10. 2024Adaptive Discounting of Training Time Attackscited by 0
  11. 2025Optimally Installing Strict Equilibriacited by 0
  12. 2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learningcited by 0
  13. 2026When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPscited by 0
  14. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 0