mlsanityData to 7 Oct 2026

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

arXiv 1911.10635 ↗First posted 24 Nov 2019cs.LG cs.AI cs.MAcited by 555

Not on the map. Why?

Cited by 46 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered great success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Though empirically successful, theoretical foundations for MARL are relatively lacking in the literature. In this chapter, we provide a selective overview of MARL, with focus on algorithms backed by theoretical analysis. More specifically, we review the theoretical results of MARL algorithms mainly within two representative frameworks, Markov/stochastic games and extensive-form games, in accordance with the types of tasks they address, i.e., fully cooperative, fully competitive, and a mix of the two. We also introduce several significant but challenging applications of these algorithms. Orthogonal to the existing reviews on MARL, we highlight several new angles and taxonomies of MARL theory, including learning in extensive-form games, decentralized MARL with networked agents, MARL in the mean-field regime, (non-)convergence of policy-based methods for learning in games, etc. Some of the new angles extrapolate from our own research endeavors and interests. Our overall goal with this chapter is, beyond providing an assessment of the current state of the field on the mark, to identify fruitful future research directions on theoretical studies of MARL. We expect this chapter to serve as continuing stimulus for researchers interested in working on this exciting while challenging topic.

Key facts. Cited by 555 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 69; 2022: 101; 2023: 115; 2024: 85; 2025: 106; 2026: 79.

555 papers.

Sort:

By area:

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  2. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  3. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  4. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 103
  5. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  6. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  7. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  8. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  9. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
  10. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  11. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 81
  12. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
38 more
  1. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
  2. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  3. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 60
  4. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  5. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 56
  6. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
  7. 2021A Survey of Machine Learning for Computer Architecture and Systemscited by 53
  8. 2021Decentralized Q-Learning in Zero-sum Markov Gamescited by 53
  9. 2021Faster Single-loop Algorithms for Minimax Optimization without Strong Concavitycited by 47
  10. 2022The Complexity of Markov Equilibrium in Stochastic Gamescited by 45
  11. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  12. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  13. 2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
  14. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  15. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 40
  16. 2021Discrete-Valued Neural Communicationcited by 39
  17. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  18. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  19. 2025Multi-Agent Coordination across Diverse Applications: A Surveycited by 37
  20. 2022Multi-agent Dynamic Algorithm Configurationcited by 36
  21. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  22. 2021Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Surveycited by 34
  23. 2023Heterogeneous Multi-Robot Reinforcement Learningcited by 34
  24. 2021Gradient play in stochastic games: stationary points, convergence, and sample complexitycited by 32
  25. 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 31
  26. 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 31
  27. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  28. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  29. 2023Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximationcited by 29
  30. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  31. 2021Multi-Agent Reinforcement Learning with Temporal Logic Specificationscited by 27
  32. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 27
  33. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  34. 2021The Confluence of Networks, Games and Learningcited by 26
  35. 2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 26
  36. 2023Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximationcited by 26
  37. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  38. 2022Performative Reinforcement Learningcited by 25

50 most cited of 555 All 555 (JSON)