# A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Games

arXiv: https://arxiv.org/abs/2509.03682 · HTML: https://mlsanity.com/paper/2509.03682/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 88 references that are in the map.
- First posted on arXiv 3 Sep 2025.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications](https://mlsanity.com/paper/1812.11794/) (2018)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games](https://mlsanity.com/paper/1703.10069/) (2017)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.07971/) (2020)
- [ViZDoom Competitions: Playing Doom from Pixels](https://mlsanity.com/paper/1809.03470/) (2018)
- [Deep Learning for Video Game Playing](https://mlsanity.com/paper/1708.07902/) (2017)
- [Deep Reinforcement Learning for Navigation in AAA Video Games](https://mlsanity.com/paper/2011.04764/) (2020)
- [The Multi-Agent Reinforcement Learning in Malm"O (MARL"O) Competition](https://mlsanity.com/paper/1901.08129/) (2019)
- [Creating Pro-Level AI for a Real-Time Fighting Game Using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.03821/) (2019)
- [Hierarchical Macro Strategy Model for MOBA Game AI](https://mlsanity.com/paper/1812.07887/) (2018)

## Built on by (most cited first, 9 of 9)

- [Latent Collaboration in Multi-Agent Systems](https://mlsanity.com/paper/2511.20639/) (2025)
- [EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems](https://mlsanity.com/paper/2510.13220/) (2025)
- [MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learning](https://mlsanity.com/paper/2512.13636/) (2025)
- [MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.06623/) (2026)
- [PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training](https://mlsanity.com/paper/2509.06053/) (2025)
- [Integrating Lagrangian Neural Networks into the Dyna Framework for Reinforcement Learning](https://mlsanity.com/paper/2603.08468/) (2026)
- [Learning to Play Blackjack: A Curriculum Learning Perspective](https://mlsanity.com/paper/2604.00076/) (2026)
- [Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment](https://mlsanity.com/paper/2604.15757/) (2026)
- [Chess on Ice: Curling Tactical Decision-Making via Backward Induction and Deep Reinforcement Learning](https://mlsanity.com/paper/2608.02379/) (2026)

## Abstract (arXiv)

Recent advancements in multi-agent reinforcement learning (MARL) have demonstrated its application potential in modern games. Beginning with foundational work and progressing to landmark achievements such as AlphaStar in StarCraft II and OpenAI Five in Dota 2, MARL has proven capable of achieving superhuman performance across diverse game environments through techniques like self-play, supervised learning, and deep reinforcement learning. With its growing impact, a comprehensive review has become increasingly important in this field. This paper aims to provide a thorough examination of MARL's application from turn-based two-agent games to real-time multi-agent video games including popular genres such as Sports games, First-Person Shooter (FPS) games, Real-Time Strategy (RTS) games and Multiplayer Online Battle Arena (MOBA) games. We further analyze critical challenges posed by MARL in video games, including nonstationary, partial observability, sparse rewards, team coordination, and scalability, and highlight successful implementations in games like Rocket League, Minecraft, Quake III Arena, StarCraft II, Dota 2, Honor of Kings, etc. This paper offers insights into MARL in video game AI systems, proposes a novel method to estimate game complexity, and suggests future research directions to advance MARL and its applications in game development, inspiring further innovation in this rapidly evolving field.