# On the Pitfalls of Measuring Emergent Communication

arXiv: https://arxiv.org/abs/1903.05168 · HTML: https://mlsanity.com/paper/1903.05168/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2019.

## Built on by (most cited first, 30 of 71)

- [A Survey of Multi-Agent Deep Reinforcement Learning with Communication](https://mlsanity.com/paper/2203.08975/) (2022)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi](https://mlsanity.com/paper/2107.07630/) (2021)
- [Learning to Ground Multi-Agent Communication with Autoencoders](https://mlsanity.com/paper/2110.15349/) (2021)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [Learning to Draw: Emergent Communication through Sketching](https://mlsanity.com/paper/2106.02067/) (2021)
- [Quasi-Equivalence Discovery for Zero-Shot Emergent Communication](https://mlsanity.com/paper/2103.08067/) (2021)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [Emergent Communication under Competition](https://mlsanity.com/paper/2101.10276/) (2021)
- [Multi-Agent Path Finding with Prioritized Communication Learning](https://mlsanity.com/paper/2202.03634/) (2022)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning](https://mlsanity.com/paper/2102.12550/) (2021)
- [Dynamic population-based meta-learning for multi-agent communication with natural language](https://mlsanity.com/paper/2110.14241/) (2021)
- [Emergent Communication in Multi-Agent Reinforcement Learning for Future Wireless Networks](https://mlsanity.com/paper/2309.06021/) (2023)
- [Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control](https://mlsanity.com/paper/2310.02435/) (2023)
- [More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration](https://mlsanity.com/paper/2604.07821/) (2026)
- [Multi-Agent Curricula and Emergent Implicit Signaling](https://mlsanity.com/paper/2106.11156/) (2021)
- [An Analysis of Discretization Methods for Communication Learning with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.05669/) (2022)
- [Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generation](https://mlsanity.com/paper/2305.05116/) (2023)
- [The emergence of visual semantics through communication games](https://mlsanity.com/paper/2101.10253/) (2021)
- [Decentralized Collective World Model for Emergent Communication and Coordination](https://mlsanity.com/paper/2504.03353/) (2025)
- [In-Context Reinforcement Learning via Communicative World Models](https://mlsanity.com/paper/2508.06659/) (2025)
- [Do Latent Channels Actually Communicate? A Causal Audit of Latent Multi-Agent LLM](https://mlsanity.com/paper/2607.26773/) (2026)
- [Learning to Communicate with Strangers via Channel Randomisation Methods](https://mlsanity.com/paper/2104.09557/) (2021)
- [Over-communicate no more: Situated RL agents learn concise communication protocols](https://mlsanity.com/paper/2211.01480/) (2022)
- [From Signals to Structure: How Memory Architecture Drives Language Emergence in LLM Agents](https://mlsanity.com/paper/2607.00233/) (2026)

## Abstract (arXiv)

How do we know if communication is emerging in a multi-agent system? The vast majority of recent papers on emergent communication show that adding a communication channel leads to an increase in reward or task success. This is a useful indicator, but provides only a coarse measure of the agent's learned communication abilities. As we move towards more complex environments, it becomes imperative to have a set of finer tools that allow qualitative and quantitative insights into the emergence of communication. This may be especially useful to allow humans to monitor agents' behaviour, whether for fault detection, assessing performance, or even building trust. In this paper, we examine a few intuitive existing metrics for measuring communication, and show that they can be misleading. Specifically, by training deep reinforcement learning agents to play simple matrix games augmented with a communication channel, we find a scenario where agents appear to communicate (their messages provide information about their subsequent action), and yet the messages do not impact the environment or other agent in any way. We explain this phenomenon using ablation studies and by visualizing the representations of the learned policies. We also survey some commonly used metrics for measuring emergent communication, and provide recommendations as to when these metrics should be used.