# Policy Diversity for Cooperative Agents

arXiv: https://arxiv.org/abs/2308.14308 · HTML: https://mlsanity.com/paper/2308.14308/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Off-Policy Multi-Agent Decomposed Policy Gradients](https://mlsanity.com/paper/2007.12322/) (2020)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policies](https://mlsanity.com/paper/1906.00088/) (2019)
- [Hierarchical Macro Strategy Model for MOBA Game AI](https://mlsanity.com/paper/1812.07887/) (2018)
- [CCPT: Automatic Gameplay Testing and Validation with Curiosity-Conditioned Proximal Trajectories](https://mlsanity.com/paper/2202.10057/) (2022)
- [Hierarchical Reinforcement Learning for Multi-agent MOBA Game](https://mlsanity.com/paper/1901.08004/) (2019)
- [Regularized Soft Actor-Critic for Behavior Transfer Learning](https://mlsanity.com/paper/2209.13224/) (2022)

## Built on by (most cited first, 2 of 2)

- [Measuring Diversity of Game Scenarios](https://mlsanity.com/paper/2404.15192/) (2024)
- [Events as Triggers for Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.12388/) (2026)

## Abstract (arXiv)

Standard cooperative multi-agent reinforcement learning (MARL) methods aim to find the optimal team cooperative policy to complete a task. However there may exist multiple different ways of cooperating, which usually are very needed by domain experts. Therefore, identifying a set of significantly different policies can alleviate the task complexity for them. Unfortunately, there is a general lack of effective policy diversity approaches specifically designed for the multi-agent domain. In this work, we propose a method called Moment-Matching Policy Diversity to alleviate this problem. This method can generate different team policies to varying degrees by formalizing the difference between team policies as the difference in actions of selected agents in different policies. Theoretically, we show that our method is a simple way to implement a constrained optimization problem that regularizes the difference between two trajectory distributions by using the maximum mean discrepancy. The effectiveness of our approach is demonstrated on a challenging team-based shooter.