# A Review of Cooperation in Multi-agent Learning

arXiv: https://arxiv.org/abs/2312.05162 · HTML: https://mlsanity.com/paper/2312.05162/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 90 of its 146 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 30 of 90)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)

## Built on by (most cited first, 29 of 29)

- [Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2410.02506/) (2024)
- [Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game](https://mlsanity.com/paper/2312.17515/) (2023)
- [POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding](https://mlsanity.com/paper/2407.14931/) (2024)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [Intelligent AI Delegation](https://mlsanity.com/paper/2602.11865/) (2026)
- [Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2602.15198/) (2026)
- [Group size effects and collective misalignment in LLM multi-agent systems](https://mlsanity.com/paper/2510.22422/) (2025)
- [CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas](https://mlsanity.com/paper/2604.15267/) (2026)
- [M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality](https://mlsanity.com/paper/2503.02077/) (2025)
- [Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Models](https://mlsanity.com/paper/2407.12532/) (2024)
- [Evaluating Collective Behaviour of Hundreds of LLM Agents](https://mlsanity.com/paper/2602.16662/) (2026)
- [Aligning Individual and Collective Objectives in Multi-Agent Cooperation](https://mlsanity.com/paper/2402.12416/) (2024)
- [DSADF: Thinking Fast and Slow for Decision Making](https://mlsanity.com/paper/2505.08189/) (2025)
- [Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration](https://mlsanity.com/paper/2404.03869/) (2024)
- [Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games](https://mlsanity.com/paper/2410.07863/) (2024)
- [Emergent Risks in Generative Multi-Agent Systems](https://mlsanity.com/paper/2603.27771/) (2026)
- [Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperation](https://mlsanity.com/paper/2405.19328/) (2024)
- [Enabling Multi-Robot Collaboration from Single-Human Guidance](https://mlsanity.com/paper/2409.19831/) (2024)
- [ROMAS: A Role-Based Multi-Agent System for Database monitoring and Planning](https://mlsanity.com/paper/2412.13520/) (2024)
- [Cooperation and Control in Delegation Games](https://mlsanity.com/paper/2402.15821/) (2024)
- [Incentives to Build Houses, Trade Houses, or Trade House Building Skills in Simulated Worlds under Various Governing Systems or Institutions: Comparing Multi-agent Reinforcement Learning to Generative Agent-based Model](https://mlsanity.com/paper/2411.17724/) (2024)
- [TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient](https://mlsanity.com/paper/2312.15667/) (2023)
- [Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.02654/) (2024)
- [The AI in the Mirror: LLM Self-Recognition in an Iterated Public Goods Game](https://mlsanity.com/paper/2508.18467/) (2025)
- [Environment Complexity and Nash Equilibria in a Sequential Social Dilemma](https://mlsanity.com/paper/2408.02148/) (2024)
- [Explaining Strategic Decisions in Multi-Agent Reinforcement Learning for Aerial Combat Tactics](https://mlsanity.com/paper/2505.11311/) (2025)
- [Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.15652/) (2025)
- [In Trust We Survive: Emergent Trust Learning](https://mlsanity.com/paper/2603.17564/) (2026)
- [Certifying cooperation: a novel approach to cooperative multi-agent task generation](https://mlsanity.com/paper/2609.06586/) (2026)

## Abstract (arXiv)

Cooperation in multi-agent learning (MAL) is a topic at the intersection of numerous disciplines, including game theory, economics, social sciences, and evolutionary biology. Research in this area aims to understand both how agents can coordinate effectively when goals are aligned and how they may cooperate in settings where gains from working together are possible but possibilities for conflict abound. In this paper we provide an overview of the fundamental concepts, problem settings and algorithms of multi-agent learning. This encompasses reinforcement learning, multi-agent sequential decision-making, challenges associated with multi-agent cooperation, and a comprehensive review of recent progress, along with an evaluation of relevant metrics. Finally we discuss open challenges in the field with the aim of inspiring new avenues for research.