# An Analysis of Discretization Methods for Communication Learning with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.05669 · HTML: https://mlsanity.com/paper/2204.05669/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 11 of 11)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [On the Pitfalls of Measuring Emergent Communication](https://mlsanity.com/paper/1903.05168/) (2019)
- [Learning to Ground Multi-Agent Communication with Autoencoders](https://mlsanity.com/paper/2110.15349/) (2021)
- [Learning to Communicate Using Counterfactual Reasoning](https://mlsanity.com/paper/2006.07200/) (2020)

## Built on by (most cited first, 6 of 6)

- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [Emergent Quantized Communication](https://mlsanity.com/paper/2211.02412/) (2022)
- [ClusterComm: Discrete Communication in Decentralized MARL using Internal Representation Clustering](https://mlsanity.com/paper/2401.03504/) (2024)
- [CtD: Composition through Decomposition in Emergent Communication](https://mlsanity.com/paper/2601.10169/) (2026)
- [An In-Depth Analysis of Discretization Methods for Communication Learning using Backpropagation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.04938/) (2023)
- [Emergence, Not Bandwidth: Physical Coupling and the Limits of Learned Multi-Agent Communication](https://mlsanity.com/paper/2609.34373/) (2026)

## Abstract (arXiv)

Communication is crucial in multi-agent reinforcement learning when agents are not able to observe the full state of the environment. The most common approach to allow learned communication between agents is the use of a differentiable communication channel that allows gradients to flow between agents as a form of feedback. However, this is challenging when we want to use discrete messages to reduce the message size since gradients cannot flow through a discrete communication channel. Previous work proposed methods to deal with this problem. However, these methods are tested in different communication learning architectures and environments, making it hard to compare them. In this paper, we compare several state-of-the-art discretization methods as well as two methods that have not been used for communication learning before. We do this comparison in the context of communication learning using gradients from other agents and perform tests on several environments. Our results show that none of the methods is best in all environments. The best choice in discretization method greatly depends on the environment. However, the discretize regularize unit (DRU), straight through DRU and the straight through gumbel softmax show the most consistent results across all the tested environments. Therefore, these methods prove to be the best choice for general use while the straight through estimator and the gumbel softmax may provide better results in specific environments but fail completely in others.