# Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.11438 · HTML: https://mlsanity.com/paper/2006.11438/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 30 of 60)

- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Context-Aware Sparse Deep Coordination Graphs](https://mlsanity.com/paper/2106.02886/) (2021)
- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Making](https://mlsanity.com/paper/2411.03865/) (2024)
- [Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2404.10976/) (2024)
- [Self-Organized Polynomial-Time Coordination Graphs](https://mlsanity.com/paper/2112.03547/) (2021)
- [Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning](https://mlsanity.com/paper/2102.12550/) (2021)
- [A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.01221/) (2022)
- [Non-Linear Coordination Graphs](https://mlsanity.com/paper/2211.08404/) (2022)
- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing](https://mlsanity.com/paper/2103.01009/) (2021)
- [Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention](https://mlsanity.com/paper/2304.12653/) (2023)
- [Inferring Latent Temporal Sparse Coordination Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.19253/) (2024)
- [Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.17540/) (2022)
- [Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigation](https://mlsanity.com/paper/2302.04094/) (2023)
- [Deep Meta Coordination Graphs for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2502.04028/) (2025)
- [Scalable Anytime Planning for Multi-Agent MDPs](https://mlsanity.com/paper/2101.04788/) (2021)
- [Dealing with Non-Stationarity in MARL via Trust-Region Decomposition](https://mlsanity.com/paper/2102.10616/) (2021)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2401.04934/) (2024)
- [AgentMixer: Multi-Agent Correlated Policy Factorization](https://mlsanity.com/paper/2401.08728/) (2024)
- [Attention Graph for Multi-Robot Social Navigation with Deep Reinforcement Learning](https://mlsanity.com/paper/2401.17914/) (2024)
- [Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.16606/) (2025)
- [Evaluating Generalization and Transfer Capacity of Multi-Agent Reinforcement Learning Across Variable Number of Agents](https://mlsanity.com/paper/2111.14177/) (2021)
- [Scaling Team Coordination on Graphs with Reinforcement Learning](https://mlsanity.com/paper/2403.05787/) (2024)
- [Bandwidth-constrained Variational Message Encoding for Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2512.11179/) (2025)
- [Universally Expressive Communication in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.06758/) (2022)
- [Subequivariant Reinforcement Learning Framework for Coordinated Motion Control](https://mlsanity.com/paper/2403.15100/) (2024)
- [Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.23604/) (2025)

## Abstract (arXiv)

Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural network based module to learn to implicitly reason about the joint actions or values. DICG allows learning the tradeoff between full centralization and decentralization via standard actor-critic methods to significantly improve coordination for domains with large number of agents. We apply DICG to both centralized-training-centralized-execution and centralized-training-decentralized-execution regimes. We demonstrate that DICG solves the relative overgeneralization pathology in predatory-prey tasks as well as outperforms various MARL baselines on the challenging StarCraft II Multi-agent Challenge (SMAC) and traffic junction environments.