mlsanityData to 14 Oct 2026

Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning

arXiv 2510.11824 ↗First posted 13 Oct 2025cs.MA cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

In cooperative Multi-Agent Reinforcement Learning (MARL), it is a common practice to tune hyperparameters in ideal simulated environments to maximize cooperative performance. However, policies tuned for cooperation often fail to maintain robustness and resilience under real-world uncertainties. Building trustworthy MARL systems requires a deep understanding of robustness, which ensures stability under uncertainties, and resilience, the ability to recover from disruptions--a concept extensively studied in control systems but largely overlooked in MARL. In this paper, we present a large-scale empirical study comprising over 82,620 experiments to evaluate cooperation, robustness, and resilience in MARL across 4 real-world environments, 13 uncertainty types, and 15 hyperparameters. Our key findings are: (1) Under mild uncertainty, optimizing cooperation improves robustness and resilience, but this link weakens as perturbations intensify. Robustness and resilience also varies by algorithm and uncertainty type. (2) Robustness and resilience do not generalize across uncertainty modalities or agent scopes: policies robust to action noise for all agents may fail under observation noise on a single agent. (3) Hyperparameter tuning is critical for trustworthy MARL: surprisingly, standard practices like parameter sharing, GAE, and PopArt can hurt robustness, while early stopping, high critic learning rates, and Leaky ReLU consistently help. By optimizing hyperparameters only, we observe substantial improvement in cooperation, robustness and resilience across all MARL backbones, with the phenomenon also generalizing to robust MARL methods across these backbones. Code and results available at https://github.com/BUAA-TrustworthyMARL/adv_marl_benchmark .

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 57 of its 97 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

57 of 97 references.

References · 57

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  5. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  6. 2019The StarCraft Multi-Agent Challengecited by 597
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2017Robust Adversarial Reinforcement Learningcited by 451
  9. 2017Adversarial Attacks on Neural Network Policiescited by 320
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  11. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  12. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
38 more
  1. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  2. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  3. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  4. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  5. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  6. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  7. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  8. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  9. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  10. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  11. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  12. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
  13. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  14. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  15. 2017Delving into adversarial attacks on deep policiescited by 104
  16. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  17. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
  18. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  19. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  20. 2020Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learningcited by 61
  21. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  22. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 54
  23. 2020Multi-agent Reinforcement Learning for Networked System Controlcited by 53
  24. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 53
  25. 2021Scientific multi-agent reinforcement learning for wall-models of turbulent flowscited by 52
  26. 2020On the Robustness of Cooperative Multi-Agent Reinforcement Learningcited by 39
  27. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  28. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 36
  29. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 34
  30. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  31. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 25
  32. 2022Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learningcited by 21
  33. 2020Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDERcited by 19
  34. 2021Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learningcited by 19
  35. 2022Robust Policy Learning over Multiple Uncertainty Setscited by 18
  36. 2023Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Gamecited by 18
  37. 2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
  38. 2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 17

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
  2. 2024Safe and Robust Reinforcement Learning: Principles and Practice
  3. 2024Collaborative Adaptation for Recovery from Unforeseen Malfunctions in Discrete and Continuous MARL Domains ↗
  4. 2025Robust and Efficient Communication in Multi-Agent Reinforcement Learning
  5. 2025Optimism as Risk-Seeking in Multi-Agent Reinforcement Learning
  6. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
  7. 2025Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics
  8. 2025Sample-Efficient Distributionally Robust Multi-Agent Reinforcement Learning via Online Interaction