# Prioritized League Reinforcement Learning for Large-Scale Heterogeneous Multiagent Systems

arXiv: https://arxiv.org/abs/2403.18057 · HTML: https://mlsanity.com/paper/2403.18057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.10800/) (2020)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity](https://mlsanity.com/paper/1707.09183/) (2017)
- [MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence](https://mlsanity.com/paper/1712.00600/) (2017)
- [Value-Decomposition Multi-Agent Actor-Critics](https://mlsanity.com/paper/2007.12306/) (2020)
- [Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks](https://mlsanity.com/paper/1712.00162/) (2017)
- [Heterogeneous Attentions for Solving Pickup and Delivery Problem via Deep Reinforcement Learning](https://mlsanity.com/paper/2110.02634/) (2021)
- [Heterogeneous Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2301.07137/) (2023)
- [On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)](https://mlsanity.com/paper/2109.04024/) (2021)
- [Deep Reinforcement Learning for Solving the Heterogeneous Capacitated Vehicle Routing Problem](https://mlsanity.com/paper/2110.02629/) (2021)
- [SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II](https://mlsanity.com/paper/2012.13169/) (2020)
- [Heterogeneous Multi-Agent Reinforcement Learning for Unknown Environment Mapping](https://mlsanity.com/paper/2010.02663/) (2020)
- [Natural Emergence of Heterogeneous Strategies in Artificially Intelligent Competitive Teams](https://mlsanity.com/paper/2007.03102/) (2020)
- [Cooperative Heterogeneous Deep Reinforcement Learning](https://mlsanity.com/paper/2011.00791/) (2020)
- [Concentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems](https://mlsanity.com/paper/2203.06416/) (2022)
- [Learning Heterogeneous Agent Cooperation via Multiagent League Training](https://mlsanity.com/paper/2211.11616/) (2022)

## Abstract (arXiv)

Large-scale heterogeneous multiagent systems feature various realistic factors in the real world, such as agents with diverse abilities and overall system cost. In comparison to homogeneous systems, heterogeneous systems offer significant practical advantages. Nonetheless, they also present challenges for multiagent reinforcement learning, including addressing the non-stationary problem and managing an imbalanced number of agents with different types. We propose a Prioritized Heterogeneous League Reinforcement Learning (PHLRL) method to address large-scale heterogeneous cooperation problems. PHLRL maintains a record of various policies that agents have explored during their training and establishes a heterogeneous league consisting of diverse policies to aid in future policy optimization. Furthermore, we design a prioritized policy gradient approach to compensate for the gap caused by differences in the number of different types of agents. Next, we use Unreal Engine to design a large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent Operation (LSMO), which is a complex two-team competition scenario that requires collaboration from both ground and airborne agents. We use experiments to show that PHLRL outperforms state-of-the-art methods, including QTRAN and QPLEX in LSMO.