mlsanityData to 14 Oct 2026

What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?

arXiv 2212.02705 ↗First posted 6 Dec 2022cs.AI cs.GT cs.MAcited by 22

Not on the map. Why?

Abstract (arXiv)

Various methods for Multi-Agent Reinforcement Learning (MARL) have been developed with the assumption that agents' policies are based on accurate state information. However, policies learned through Deep Reinforcement Learning (DRL) are susceptible to adversarial state perturbation attacks. In this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to investigate different solution concepts of MARL under state uncertainties. Our analysis shows that the commonly used solution concepts of optimal agent policy and robust Nash equilibrium do not always exist in SAMGs. To circumvent this difficulty, we consider a new solution concept called robust agent policy, where agents aim to maximize the worst-case expected state value. We prove the existence of robust agent policy for finite state and finite action SAMGs. Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C) algorithm to learn robust policies for MARL agents under state uncertainties. Our experiments demonstrate that our algorithm outperforms existing methods when faced with state perturbations and greatly improves the robustness of MARL policies. Our code is public on https://songyanghan.github.io/what_is_solution/.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 37 of its 72 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

37 of 72 references.

References · 37

  1. 2016Adversarial examples in the physical worldcited by 1,769
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  4. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  5. 2017Robust Adversarial Reinforcement Learningcited by 451
  6. 2016Learning Multiagent Communication with Backpropagationcited by 366
  7. 2017Adversarial Attacks on Neural Network Policiescited by 320
  8. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  9. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  11. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 231
  12. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
25 more
  1. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 209
  2. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  3. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  4. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  5. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  6. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 114
  7. 2017Delving into adversarial attacks on deep policiescited by 104
  8. 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
  9. 2020Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Rewardcited by 50
  10. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 48
  11. 2017Guided Deep Reinforcement Learning for Swarm Systemscited by 44
  12. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
  13. 2020On the Robustness of Cooperative Multi-Agent Reinforcement Learningcited by 39
  14. 2020Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advancescited by 38
  15. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  16. 2019Characterizing Attacks on Deep Reinforcement Learningcited by 31
  17. 2021SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learningcited by 27
  18. 2020Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learningcited by 24
  19. 2020SS-SFDA : Self-Supervised Source-Free Domain Adaptation for Road Segmentation in Hazardous Environmentscited by 18
  20. 2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
  21. 2022Stable and Efficient Shapley Value-Based Reward Reallocation for Multi-Agent Reinforcement Learning of Autonomous Vehiclescited by 11
  22. 2020FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesiscited by 10
  23. 2022Certifiably Robust Policy Learning against Adversarial Communication in Multi-agent Systemscited by 7
  24. 2020Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communicationcited by 6
  25. 2020A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 2; 2025: 5; 2026: 3.

22 papers.

Sort:

By area:

  1. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  2. 2023Collaborative Multi-Object Tracking with Conformal Uncertainty Propagationcited by 27
  3. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  4. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  5. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 25
  6. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 13
  7. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  8. 2023Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agentscited by 7
  9. 2023PIMbot: Policy and Incentive Manipulation for Multi-Robot Reinforcement Learning in Social Dilemmascited by 6
  10. 2023Robust Electric Vehicle Balancing of Autonomous Mobility-On-Demand System: A Multi-Agent Reinforcement Learning Approachcited by 6
  11. 2023Robust Multi-Agent Reinforcement Learning by Mutual Information Regularizationcited by 6
  12. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
10 more
  1. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  2. 2024Minimax-Optimal Multi-Agent Robust Reinforcement Learningcited by 5
  3. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  4. 2023Enhancing the Robustness of QMIX against State-adversarial Attackscited by 3
  5. 2025STORI: A Benchmark and Taxonomy for Stochastic Environmentscited by 2
  6. 2023Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learningcited by 1
  7. 2026Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learningcited by 1
  8. 2026PIMbot: A Self-Adaptive Attack Framework for Adversarial Manipulation of Multi-Robot Reinforcement Learningcited by 1
  9. 2025Robust and Safe Multi-Agent Reinforcement Learning with Communication for Autonomous Vehicles: From Simulation to Hardwarecited by 0
  10. 2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0