# Multi-agent Reinforcement Learning: A Comprehensive Survey

arXiv: https://arxiv.org/abs/2312.10256 · HTML: https://mlsanity.com/paper/2312.10256/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 130 of its 244 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Builds on (most cited first, 30 of 130)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)

## Built on by (most cited first, 30 of 33)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning](https://mlsanity.com/paper/2505.23399/) (2025)
- [LLM-Powered Decentralized Generative Agents with Adaptive Hierarchical Knowledge Graph for Cooperative Planning](https://mlsanity.com/paper/2502.05453/) (2025)
- [AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System Need](https://mlsanity.com/paper/2506.15451/) (2025)
- [Generative AI collective behavior needs an interactionist paradigm](https://mlsanity.com/paper/2601.10567/) (2026)
- [Evaluating LLMs in Open-Source Games](https://mlsanity.com/paper/2512.00371/) (2025)
- [CANDID DAC: Leveraging Coupled Action Dimensions with Importance Differences in DAC](https://mlsanity.com/paper/2407.05789/) (2024)
- [Representation Learning For Efficient Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.02890/) (2024)
- [Skill-Aligned Fairness in Multi-Agent Learning for Collaboration in Healthcare](https://mlsanity.com/paper/2508.18708/) (2025)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [CRAFT: Grounded Multi-Agent Coordination Under Partial Information](https://mlsanity.com/paper/2603.25268/) (2026)
- [Building Better Environments for Autonomous Cyber Defence](https://mlsanity.com/paper/2604.08805/) (2026)
- [Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.23604/) (2025)
- [Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Survey](https://mlsanity.com/paper/2508.20315/) (2025)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [Multi-agent Auto-Bidding with Latent Graph Diffusion Models](https://mlsanity.com/paper/2503.05805/) (2025)
- [Multi-Agent Reinforcement Learning in Cybersecurity: From Fundamentals to Applications](https://mlsanity.com/paper/2505.19837/) (2025)
- [Safe Heterogeneous Multi-Agent RL with Communication Regularization for Coordinated Target Acquisition](https://mlsanity.com/paper/2601.08327/) (2026)
- [MACRO-LLM: LLM-Empowered Multi-Agent Collaborative Reasoning under Spatiotemporal Partial Observability](https://mlsanity.com/paper/2601.09295/) (2026)
- [Local-Canonicalization Equivariant Graph Neural Networks for Sample-Efficient and Generalizable Swarm Robot Control](https://mlsanity.com/paper/2509.14431/) (2025)
- [MAGIC-MASK: Multi-Agent Guided Inter-Agent Collaboration with Mask-Based Explainability for Reinforcement Learning](https://mlsanity.com/paper/2510.00274/) (2025)
- [Distributed Area Coverage with High Altitude Balloons Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.03823/) (2025)
- [Persona-based Multi-Agent Collaboration for Brainstorming](https://mlsanity.com/paper/2512.04488/) (2025)
- [MaRCA: Multi-Agent Reinforcement Learning for Dynamic Computation Allocation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2512.24325/) (2025)
- [Co-jump: Cooperative Jumping with Quadrupedal Robots via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.10514/) (2026)
- [Subjective-Graph LLM Agents for Simulating Uncertainty in Classroom Social Perception](https://mlsanity.com/paper/2603.20750/) (2026)
- [Multi-Agent Systems: From Classical Paradigms to Large Foundation Model-Enabled Futures](https://mlsanity.com/paper/2604.18133/) (2026)

## Abstract (arXiv)

Multi-agent systems (MAS) are widely prevalent and crucially important in numerous real-world applications, where multiple agents must make decisions to achieve their objectives in a shared environment. Despite their ubiquity, the development of intelligent decision-making agents in MAS poses several open challenges to their effective implementation. This survey examines these challenges, placing an emphasis on studying seminal concepts from game theory (GT) and machine learning (ML) and connecting them to recent advancements in multi-agent reinforcement learning (MARL), i.e. the research of data-driven decision-making within MAS. Therefore, the objective of this survey is to provide a comprehensive perspective along the various dimensions of MARL, shedding light on the unique opportunities that are presented in MARL applications while highlighting the inherent challenges that accompany this potential. Therefore, we hope that our work will not only contribute to the field by analyzing the current landscape of MARL but also motivate future directions with insights for deeper integration of concepts from related domains of GT and ML. With this in mind, this work delves into a detailed exploration of recent and past efforts of MARL and its related fields and describes prior solutions that were proposed and their limitations, as well as their applications.