# Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication

arXiv: https://arxiv.org/abs/2004.00470 · HTML: https://mlsanity.com/paper/2004.00470/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 12 of 12)

- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applications](https://mlsanity.com/paper/2206.07922/) (2022)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Counterfactual Learning on Graphs: A Survey](https://mlsanity.com/paper/2304.01391/) (2023)
- [Distributed Reinforcement Learning for Robot Teams: A Review](https://mlsanity.com/paper/2204.03516/) (2022)
- [MAGNNETO: A Graph Neural Network-based Multi-Agent system for Traffic Engineering](https://mlsanity.com/paper/2303.18157/) (2023)
- [Is Machine Learning Ready for Traffic Engineering Optimization?](https://mlsanity.com/paper/2109.01445/) (2021)
- [Deep Reinforcement Learning for Autonomous Cyber Defence: A Survey](https://mlsanity.com/paper/2310.07745/) (2023)
- [Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2302.10825/) (2023)
- [RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2206.01207/) (2022)
- [QMNet: Importance-Aware Message Exchange for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.09051/) (2023)
- [Interference-Aware K-Step Reachable Communication in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2603.15054/) (2026)

## Abstract (arXiv)

We consider a fully cooperative multi-agent system where agents cooperate to maximize a system's utility in a partial-observable environment. We propose that multi-agent systems must have the ability to (1) communicate and understand the inter-plays between agents and (2) correctly distribute rewards based on an individual agent's contribution. In contrast, most work in this setting considers only one of the above abilities. In this study, we develop an architecture that allows for communication among agents and tailors the system's reward for each individual agent. Our architecture represents agent communication through graph convolution and applies an existing credit assignment structure, counterfactual multi-agent policy gradient (COMA), to assist agents to learn communication by back-propagation. The flexibility of the graph structure enables our method to be applicable to a variety of multi-agent systems, e.g. dynamic systems that consist of varying numbers of agents and static systems with a fixed number of agents. We evaluate our method on a range of tasks, demonstrating the advantage of marrying communication with credit assignment. In the experiments, our proposed method yields better performance than the state-of-art methods, including COMA. Moreover, we show that the communication strategies offers us insights and interpretability of the system's cooperative policies.