# What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?

arXiv: https://arxiv.org/abs/2212.02705 · HTML: https://mlsanity.com/paper/2212.02705/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 72 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 30 of 37)

- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems](https://mlsanity.com/paper/1906.00331/) (2019)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.10800/) (2020)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)
- [Robust Deep Reinforcement Learning with Adversarial Attacks](https://mlsanity.com/paper/1712.03632/) (2017)
- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL](https://mlsanity.com/paper/2110.14555/) (2021)
- [Delving into adversarial attacks on deep policies](https://mlsanity.com/paper/1705.06452/) (2017)
- [Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks](https://mlsanity.com/paper/1701.04143/) (2017)
- [Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward](https://mlsanity.com/paper/2006.06626/) (2020)
- [Value-Decomposition Multi-Agent Actor-Critics](https://mlsanity.com/paper/2007.12306/) (2020)
- [Guided Deep Reinforcement Learning for Swarm Systems](https://mlsanity.com/paper/1709.06011/) (2017)
- [Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning](https://mlsanity.com/paper/2210.05927/) (2022)
- [On the Robustness of Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.03722/) (2020)
- [Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advances](https://mlsanity.com/paper/2006.08141/) (2020)
- [Improving Policies via Search in Cooperative Partially Observable Games](https://mlsanity.com/paper/1912.02318/) (2019)
- [Characterizing Attacks on Deep Reinforcement Learning](https://mlsanity.com/paper/1907.09470/) (2019)
- [SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2101.06005/) (2021)
- [Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/2004.06496/) (2020)

## Built on by (most cited first, 22 of 22)

- [The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2305.16589/) (2023)
- [Collaborative Multi-Object Tracking with Conformal Uncertainty Propagation](https://mlsanity.com/paper/2303.14346/) (2023)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty](https://mlsanity.com/paper/2404.18909/) (2024)
- [Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning](https://mlsanity.com/paper/2502.19652/) (2025)
- [Solving a Class of Non-Convex Minimax Optimization in Federated Learning](https://mlsanity.com/paper/2310.03613/) (2023)
- [Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents](https://mlsanity.com/paper/2306.06136/) (2023)
- [PIMbot: Policy and Incentive Manipulation for Multi-Robot Reinforcement Learning in Social Dilemmas](https://mlsanity.com/paper/2307.15944/) (2023)
- [Robust Electric Vehicle Balancing of Autonomous Mobility-On-Demand System: A Multi-Agent Reinforcement Learning Approach](https://mlsanity.com/paper/2307.16228/) (2023)
- [Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization](https://mlsanity.com/paper/2310.09833/) (2023)
- [Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.02844/) (2025)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Minimax-Optimal Multi-Agent Robust Reinforcement Learning](https://mlsanity.com/paper/2412.19873/) (2024)
- [Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality](https://mlsanity.com/paper/2508.17448/) (2025)
- [Enhancing the Robustness of QMIX against State-adversarial Attacks](https://mlsanity.com/paper/2307.00907/) (2023)
- [STORI: A Benchmark and Taxonomy for Stochastic Environments](https://mlsanity.com/paper/2509.01793/) (2025)
- [Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learning](https://mlsanity.com/paper/2310.08746/) (2023)
- [Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.18024/) (2026)
- [PIMbot: A Self-Adaptive Attack Framework for Adversarial Manipulation of Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2605.23027/) (2026)
- [Robust and Safe Multi-Agent Reinforcement Learning with Communication for Autonomous Vehicles: From Simulation to Hardware](https://mlsanity.com/paper/2506.00982/) (2025)
- [Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorization](https://mlsanity.com/paper/2602.11437/) (2026)

## Abstract (arXiv)

Various methods for Multi-Agent Reinforcement Learning (MARL) have been developed with the assumption that agents' policies are based on accurate state information. However, policies learned through Deep Reinforcement Learning (DRL) are susceptible to adversarial state perturbation attacks. In this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to investigate different solution concepts of MARL under state uncertainties. Our analysis shows that the commonly used solution concepts of optimal agent policy and robust Nash equilibrium do not always exist in SAMGs. To circumvent this difficulty, we consider a new solution concept called robust agent policy, where agents aim to maximize the worst-case expected state value. We prove the existence of robust agent policy for finite state and finite action SAMGs. Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C) algorithm to learn robust policies for MARL agents under state uncertainties. Our experiments demonstrate that our algorithm outperforms existing methods when faced with state perturbations and greatly improves the robustness of MARL policies. Our code is public on https://songyanghan.github.io/what_is_solution/.