# Multi-Agent Common Knowledge Reinforcement Learning

arXiv: https://arxiv.org/abs/1810.11702 · HTML: https://mlsanity.com/paper/1810.11702/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2018.

## Built on by (most cited first, 30 of 31)

- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning](https://mlsanity.com/paper/2102.07936/) (2021)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [Solving Common-Payoff Games with Approximate Policy Iteration](https://mlsanity.com/paper/2101.04237/) (2021)
- [From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARL](https://mlsanity.com/paper/2304.14656/) (2023)
- [A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.01221/) (2022)
- [On Centralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.14597/) (2024)
- [Common Information based Approximate State Representations in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2110.12603/) (2021)
- [Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees](https://mlsanity.com/paper/2110.05597/) (2021)
- [Generalized Beliefs for Cooperative AI](https://mlsanity.com/paper/2206.12765/) (2022)
- [A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.00451/) (2023)
- [AgentMixer: Multi-Agent Correlated Policy Factorization](https://mlsanity.com/paper/2401.08728/) (2024)
- [MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiation](https://mlsanity.com/paper/2410.14383/) (2024)
- [Causal Knowledge Transfer for Multi-Agent Reinforcement Learning in Dynamic Environments](https://mlsanity.com/paper/2507.13846/) (2025)
- [Discovering Causality for Efficient Cooperation in Multi-Agent Environments](https://mlsanity.com/paper/2306.11846/) (2023)
- [Symmetry-Breaking Augmentations for Ad Hoc Teamwork](https://mlsanity.com/paper/2402.09984/) (2024)
- [Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning](https://mlsanity.com/paper/2102.04775/) (2021)
- [A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.02430/) (2023)
- [Behavioral Universe Network (BUN): A Behavioral Information-Based Framework for Complex Systems](https://mlsanity.com/paper/2504.15146/) (2025)
- [Learning to Coordinate via Quantum Entanglement in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.08965/) (2026)
- [Causality Detection for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.14227/) (2023)
- [Multicopy Reinforcement Learning Agents](https://mlsanity.com/paper/2309.10908/) (2023)
- [FlickerFusion: Intra-trajectory Domain Generalizing Multi-Agent RL](https://mlsanity.com/paper/2410.15876/) (2024)
- [Efficient Training in Multi-Agent Reinforcement Learning: A Communication-Free Framework for the Box-Pushing Problem](https://mlsanity.com/paper/2411.12246/) (2024)
- [ICCO: Learning an Instruction-conditioned Coordinator for Language-guided Task-aligned Multi-robot Control](https://mlsanity.com/paper/2503.12122/) (2025)

## Abstract (arXiv)

Cooperative multi-agent reinforcement learning often requires decentralised policies, which severely limit the agents' ability to coordinate their behaviour. In this paper, we show that common knowledge between agents allows for complex decentralised coordination. Common knowledge arises naturally in a large number of decentralised cooperative multi-agent tasks, for example, when agents can reconstruct parts of each others' observations. Since agents an independently agree on their common knowledge, they can execute complex coordinated policies that condition on this knowledge in a fully decentralised fashion. We propose multi-agent common knowledge reinforcement learning (MACKRL), a novel stochastic actor-critic algorithm that learns a hierarchical policy tree. Higher levels in the hierarchy coordinate groups of agents by conditioning on their common knowledge, or delegate to lower levels with smaller subgroups but potentially richer common knowledge. The entire policy tree can be executed in a fully decentralised fashion. As the lowest policy tree level consists of independent policies for each agent, MACKRL reduces to independently learnt decentralised policies as a special case. We demonstrate that our method can exploit common knowledge for superior performance on complex decentralised coordination tasks, including a stochastic matrix game and challenging problems in StarCraft II unit micromanagement.