mlsanityData to 14 Oct 2026

Towards a Standardised Performance Evaluation Protocol for Cooperative MARL

arXiv 2209.10485 ↗First posted 21 Sep 2022cs.LG cs.AI cs.GLcited by 49

Not on the map. Why?

Abstract (arXiv)

Multi-agent reinforcement learning (MARL) has emerged as a useful approach to solving decentralised decision-making problems at scale. Research in the field has been growing steadily with many breakthrough algorithms proposed in recent years. In this work, we take a closer look at this rapid development with a focus on evaluation methodologies employed across a large body of research in cooperative MARL. By conducting a detailed meta-analysis of prior work, spanning 75 papers accepted for publication from 2016 to 2022, we bring to light worrying trends that put into question the true rate of progress. We further consider these trends in a wider context and take inspiration from single-agent RL literature on similar issues with recommendations that remain applicable to MARL. Combining these recommendations, with novel insights from our analysis, we propose a standardised performance evaluation protocol for cooperative MARL. We argue that such a standard protocol, if widely adopted, would greatly improve the validity and credibility of future research, make replication and reproducibility easier, as well as improve the ability of the field to accurately gauge the rate of progress over time by being able to make sound comparisons across different works. Finally, we release our meta-analysis data publicly on our project website for future research on evaluation: https://sites.google.com/view/marl-standard-protocol

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 90 of its 110 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

90 of 110 references.

References · 90

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  4. 2018Model Cards for Model Reportingcited by 1,080
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2017Deep Reinforcement Learning that Matterscited by 754
  8. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  10. 2019The StarCraft Multi-Agent Challengecited by 597
  11. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  12. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
38 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  2. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  3. 2016Learning Multiagent Communication with Backpropagationcited by 366
  4. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  5. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  6. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  7. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  8. 2018TarMAC: Targeted Multi-Agent Communicationcited by 208
  9. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  10. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  11. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  12. 2019MAVEN: Multi-Agent Variational Explorationcited by 185
  13. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
  14. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  15. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  16. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  17. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 148
  18. 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
  19. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  20. 2018Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Taskscited by 121
  21. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 117
  22. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  23. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  24. 2019Deep Coordination Graphscited by 98
  25. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  26. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 94
  27. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  28. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  29. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
  30. 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 86
  31. 2019Multi-Agent Game Abstraction via Graph Attention Neural Networkcited by 82
  32. 2019Influence-Based Multi-Agent Explorationcited by 75
  33. 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 73
  34. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  35. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 60
  36. 2020Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learningcited by 60
  37. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 58
  38. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 15; 2025: 12; 2026: 9.

49 papers.

Sort:

By area:

  1. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  2. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  3. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  4. 2022Multi-agent Dynamic Algorithm Configurationcited by 36
  5. 2023BenchMARL: Benchmarking Multi-Agent Reinforcement Learningcited by 33
  6. 2023A Review of Cooperation in Multi-agent Learningcited by 29
  7. 2024POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfindingcited by 21
  8. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 16
  9. 2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 11
  10. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  11. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  12. 2025Benchmarking Quantum Reinforcement Learningcited by 10
37 more
  1. 2023IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARLcited by 9
  2. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  3. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 8
  4. 2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performancecited by 7
  5. 2024MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learningcited by 6
  6. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 5
  7. 2024CoDreamer: Communication-Based Decentralised World Modelscited by 5
  8. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  9. 2024SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMACcited by 5
  10. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Taskscited by 5
  11. 2023Efficiently Quantifying Individual Agent Importance in Cooperative MARLcited by 4
  12. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  13. 2023Robust Multi-agent Communication via Multi-view Message Certificationcited by 3
  14. 2024Representation Learning For Efficient Deep Multi-Agent Reinforcement Learningcited by 3
  15. 2024Beyond the Boundaries of Proximal Policy Optimizationcited by 3
  16. 2025Multi-Agent Guided Policy Optimizationcited by 3
  17. 2023Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learningcited by 2
  18. 2023On Diagnostics for Understanding Agent Training Behaviour in Cooperative MARLcited by 2
  19. 2024Multi-Agent Reinforcement Learning with Selective State-Space Modelscited by 2
  20. 2025CAMAR: Continuous Actions Multi-Agent Routingcited by 2
  21. 2026Probing Dec-POMDP Reasoning in Cooperative MARLcited by 2
  22. 2023How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learningcited by 1
  23. 2025A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inferencecited by 1
  24. 2026Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Gamescited by 1
  25. 2024Attention-Driven Multi-Agent Reinforcement Learning: Enhancing Decisions with Expertise-Informed Taskscited by 0
  26. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  27. 2025Assistax: A Multi-Agent Hardware-Accelerated Reinforcement Learning Benchmark for Assistive Roboticscited by 0
  28. 2025Spatial Computing Communications for Multi-User Virtual Reality in Distributed Mobile Edge Computing Networkcited by 0
  29. 2025Intelligent Collaborative Optimization for Rubber Tyre Film Production Based on Multi-path Differentiated Clipping Proximal Policy Optimizationcited by 0
  30. 2025Semi-overlapping Multi-bandit Best Arm Identification for Sequential Support Network Learningcited by 0
  31. 2026Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environmentcited by 0
  32. 2026Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmarkcited by 0
  33. 2026Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learningcited by 0
  34. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  35. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  36. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0
  37. 2026MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RLcited by 0