Data to 7 Oct 2026

Adaptive Reward-Poisoning Attacks against Reinforcement Learning

arXiv 2003.12613 ↗First posted 27 Mar 2020cs.LG cs.AI cs.CRcited by 86

Not on the map. Why?

Abstract (arXiv)

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward rt into rt+δt at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on δt: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where δt depends only on (st,at, st+1), or adaptive where δt depends further on the RL agent's learning process at time t. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size |S|, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 17; 2023: 19; 2024: 16; 2025: 14; 2026: 6.

86 papers.

Sort:

By area:

2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 93
2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 54
2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
2021Robust Policy Gradient against Strong Data Corruptioncited by 30
2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
2021Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learningcited by 29
2023BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learningcited by 29
2021Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environmentscited by 26
2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26

38 more

2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attackscited by 25
2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 20
2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
2021Reinforcement Learning for Feedback-Enabled Cyber Resiliencecited by 18
2021Poisoning Attack against Estimating from Pairwise Comparisonscited by 14
2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
2022Admissible Policy Teaching through Reward Designcited by 12
2022A Search-Based Testing Approach for Deep Reinforcement Learning Agentscited by 12
2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
2023Optimal Attack and Defense for Reinforcement Learningcited by 12
2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
2021When Are Linear Stochastic Bandits Attackable?cited by 10
2021Iterative Teaching by Label Synthesiscited by 10
2022Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learningcited by 9
2023Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learningcited by 9
2021Game Redesign in No-regret Game Playingcited by 8
2022Iterative Teaching by Data Hallucinationcited by 8
2024Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policiescited by 8
2025Provably Robust Federated Reinforcement Learningcited by 8
2022New Challenges in Reinforcement Learning: A Survey of Security and Privacycited by 7
2024SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systemscited by 7
2023Adversarial Attacks on Adversarial Banditscited by 6
2023Local Environment Poisoning Attacks on Federated Reinforcement Learningcited by 6
2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
2022Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysiscited by 5
2022Reward Delay Attacks on Deep Reinforcement Learningcited by 5
2023Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learningcited by 5
2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performancecited by 4
2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
2024Data Poisoning Attacks on Off-Policy Policy Evaluation Methodscited by 4
2021Reward-Free Attacks in Multi-Agent Reinforcement Learningcited by 3
2021Efficient Action Poisoning Attacks on Linear Contextual Banditscited by 3
2024Exploring Fairness in Educational Data Mining in the Context of the Right to be Forgottencited by 3
2023A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agentscited by 2
2023Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertaintycited by 2
2023Adversarial Attacks on Combinatorial Multi-Armed Banditscited by 2

50 most cited of 86 All 86 (JSON)