# Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2206.01888 · HTML: https://mlsanity.com/paper/2206.01888/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 4 Jun 2022.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 30 of 31)

- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium](https://mlsanity.com/paper/2002.07066/) (2020)
- [Adaptive Reward-Poisoning Attacks against Reinforcement Learning](https://mlsanity.com/paper/2003.12613/) (2020)
- [Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning](https://mlsanity.com/paper/2003.12909/) (2020)
- [Corruption-robust exploration in episodic reinforcement learning](https://mlsanity.com/paper/1911.08689/) (2019)
- [Policy Poisoning in Batch Reinforcement Learning and Control](https://mlsanity.com/paper/1910.05821/) (2019)
- [Stochastic Linear Bandits Robust to Adversarial Attacks](https://mlsanity.com/paper/2007.03285/) (2020)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [Adversarial Attacks on Stochastic Bandits](https://mlsanity.com/paper/1810.12188/) (2018)
- [Data Poisoning Attacks on Stochastic Bandits](https://mlsanity.com/paper/1905.06494/) (2019)
- [Corruption-Robust Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06630/) (2021)
- [Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals](https://mlsanity.com/paper/1906.10571/) (2019)
- [A Model Selection Approach for Corruption Robust Reinforcement Learning](https://mlsanity.com/paper/2110.03580/) (2021)
- [Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasets](https://mlsanity.com/paper/2202.07511/) (2022)
- [Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics](https://mlsanity.com/paper/2009.00774/) (2020)
- [Adversarial Attacks on Linear Contextual Bandits](https://mlsanity.com/paper/2002.03839/) (2020)
- [Offline Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.01832/) (2021)
- [Robust Policy Gradient against Strong Data Corruption](https://mlsanity.com/paper/2102.05800/) (2021)
- [Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2110.04471/) (2021)
- [When is Offline Two-Player Zero-Sum Markov Game Solvable?](https://mlsanity.com/paper/2201.03522/) (2022)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks](https://mlsanity.com/paper/2203.08398/) (2022)
- [Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks](https://mlsanity.com/paper/2011.10824/) (2020)
- [Data Poisoning Attacks in Contextual Bandits](https://mlsanity.com/paper/1808.05760/) (2018)
- [Defense Against Reward Poisoning Attacks in Reinforcement Learning](https://mlsanity.com/paper/2102.05776/) (2021)
- [Robust Stochastic Bandit Algorithms under Probabilistic Unbounded Adversarial Attack](https://mlsanity.com/paper/2002.07214/) (2020)
- [Admissible Policy Teaching through Reward Design](https://mlsanity.com/paper/2201.02185/) (2022)
- [Saving Stochastic Bandits from Poisoning Attacks via Limited Data Verification](https://mlsanity.com/paper/2102.07711/) (2021)

## Built on by (most cited first, 20 of 20)

- [Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attacks](https://mlsanity.com/paper/2302.13851/) (2023)
- [Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2307.07670/) (2023)
- [TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents](https://mlsanity.com/paper/2404.12530/) (2024)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Corruption-Robust Offline Two-Player Zero-Sum Markov Games](https://mlsanity.com/paper/2403.07933/) (2024)
- [Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving](https://mlsanity.com/paper/2509.16950/) (2025)
- [Minimally Modifying a Markov Game to Achieve Any Nash Equilibrium and Value](https://mlsanity.com/paper/2311.00582/) (2023)
- [On the Hardness of Decentralized Multi-Agent Policy Evaluation under Byzantine Attacks](https://mlsanity.com/paper/2409.12882/) (2024)
- [Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning](https://mlsanity.com/paper/2505.20621/) (2025)
- [TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models](https://mlsanity.com/paper/2506.12815/) (2025)
- [Online Poisoning Attack Against Reinforcement Learning under Black-box Environments](https://mlsanity.com/paper/2412.00797/) (2024)
- [Enhancing Security in Deep Reinforcement Learning: A Comprehensive Survey on Adversarial Attacks and Defenses](https://mlsanity.com/paper/2510.20314/) (2025)
- [Inception: Efficiently Computable Misinformation Attacks on Markov Games](https://mlsanity.com/paper/2406.17114/) (2024)
- [Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning](https://mlsanity.com/paper/2411.13116/) (2024)
- [Efficient Preference Poisoning Attack on Offline RLHF](https://mlsanity.com/paper/2605.02495/) (2026)
- [Optimally Installing Strict Equilibria](https://mlsanity.com/paper/2503.03676/) (2025)
- [Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks](https://mlsanity.com/paper/2506.06891/) (2025)
- [Neutral Agent-based Adversarial Policy Learning against Deep Reinforcement Learning in Multi-party Open Systems](https://mlsanity.com/paper/2510.10937/) (2025)
- [Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback](https://mlsanity.com/paper/2603.28281/) (2026)
- [When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPs](https://mlsanity.com/paper/2604.10062/) (2026)

## Abstract (arXiv)

In offline multi-agent reinforcement learning (MARL), agents estimate policies from a given dataset. We study reward-poisoning attacks in this setting where an exogenous attacker modifies the rewards in the dataset before the agents see the dataset. The attacker wants to guide each agent into a nefarious target policy while minimizing the Lp norm of the reward modification. Unlike attacks on single-agent RL, we show that the attacker can install the target policy as a Markov Perfect Dominant Strategy Equilibrium (MPDSE), which rational agents are guaranteed to follow. This attack can be significantly cheaper than separate single-agent attacks. We show that the attack works on various MARL agents including uncertainty-aware learners, and we exhibit linear programs to efficiently solve the attack problem. We also study the relationship between the structure of the datasets and the minimal attack cost. Our work paves the way for studying defense in offline MARL.