# Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning

arXiv: https://arxiv.org/abs/2102.12550 · HTML: https://mlsanity.com/paper/2102.12550/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Long Short-Term Memory-Networks for Machine Reading](https://mlsanity.com/paper/1601.06733/) (2016)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks](https://mlsanity.com/paper/1812.09755/) (2018)
- [Deep Coordination Graphs](https://mlsanity.com/paper/1910.00091/) (2019)
- [Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games](https://mlsanity.com/paper/1703.10069/) (2017)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [On the Pitfalls of Measuring Emergent Communication](https://mlsanity.com/paper/1903.05168/) (2019)
- [Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2006.11438/) (2020)
- [Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data](https://mlsanity.com/paper/1803.08314/) (2018)
- [Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.13428/) (2019)

## Built on by (most cited first, 11 of 11)

- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.03358/) (2023)
- [Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translation](https://mlsanity.com/paper/2302.09368/) (2023)
- [Interpretable Learned Emergent Communication for Human-Agent Teams](https://mlsanity.com/paper/2201.07452/) (2022)
- [Towards True Lossless Sparse Communication in Multi-Agent Systems](https://mlsanity.com/paper/2212.00115/) (2022)
- [On the Role of Emergent Communication for Social Learning in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.14276/) (2023)
- [Dynamic Size Message Scheduling for Multi-Agent Communication under Limited Bandwidth](https://mlsanity.com/paper/2306.10134/) (2023)
- [Discrete Messages Improve Communication Efficiency among Isolated Intelligent Agents](https://mlsanity.com/paper/2312.15985/) (2023)
- [QMNet: Importance-Aware Message Exchange for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.09051/) (2023)
- [CoMIX: A Multi-agent Reinforcement Learning Training Architecture for Efficient Decentralized Coordination and Independent Decision-Making](https://mlsanity.com/paper/2308.10721/) (2023)
- [Learning to Cooperate and Communicate Over Imperfect Channels](https://mlsanity.com/paper/2311.14770/) (2023)

## Abstract (arXiv)

Communication is a important factor that enables agents work cooperatively in multi-agent reinforcement learning (MARL). Most previous work uses continuous message communication whose high representational capacity comes at the expense of interpretability. Allowing agents to learn their own discrete message communication protocol emerged from a variety of domains can increase the interpretability for human designers and other agents.This paper proposes a method to generate discrete messages analogous to human languages, and achieve communication by a broadcast-and-listen mechanism based on self-attention. We show that discrete message communication has performance comparable to continuous message communication but with much a much smaller vocabulary size.Furthermore, we propose an approach that allows humans to interactively send discrete messages to agents.