# When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

arXiv: https://arxiv.org/abs/2608.14559 · HTML: https://mlsanity.com/paper/2608.14559/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 8 of 8)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- [Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks](https://mlsanity.com/paper/1812.09755/) (2018)
- [T2MAC: Targeted and Trusted Multi-Agent Communication through Selective Engagement and Evidence-Driven Integration](https://mlsanity.com/paper/2401.10973/) (2024)
- [Communicating Plans, Not Percepts: Scalable Multi-Agent Coordination with Embodied World Models](https://mlsanity.com/paper/2508.02912/) (2025)

## Abstract (arXiv)

Effective communication in multi-agent reinforcement learning requires agents to decide not only textitwhat to communicate, but when? Existing approaches either communicate at every timestep or learn a binary gate through REINFORCE policy gradients citesingh2019, a high-variance signal that produces unstable and uninterpretable gating behavior. I propose a principled alternative: agents communicate only when the KL divergence between their learned belief distributions exceeds a fixed threshold. Each agent maintains a belief distribution over a latent world state computed as a softmax over its LSTM hidden state, and communicates only when belief disagreement is large enough to justify information exchange. I evaluate this approach on the Predator-Prey benchmark from IC3Net citesingh2019 across two environment sizes with 5 seeds each, and on MPE simplespread citelowe2017, comparing against IC3Net, CommNet, and an independent controller. On PP 10×10, IC3Net outperforms KL-belief at all thresholds. On the harder PP 20×20, a threshold ablation over varepsilon ∈ 0.1, 0.3, 0.5, 1.0 reveals an inverted U-shape: varepsilon=0.5 achieves 73.84 average steps and 42% success rate versus IC3Net's 75.31 steps and 31%, a gap of 1.47 steps and 11 percentage points with tighter seed variance. On MPE, the belief head improves mean reward by 12 points and reduces variance by 26× even when gating is inactive, suggesting two orthogonal contributions: principled gating when beliefs can converge, and improved latent representations that benefit coordination regardless.