# Learning to Communicate Using Counterfactual Reasoning

arXiv: https://arxiv.org/abs/2006.07200 · HTML: https://mlsanity.com/paper/2006.07200/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 9 of 9)

- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)
- [An Analysis of Discretization Methods for Communication Learning with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.05669/) (2022)
- [Mixed Cooperative-Competitive Communication Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2110.15762/) (2021)
- [Learning to Communicate with Reinforcement Learning for an Adaptive Traffic Control System](https://mlsanity.com/paper/2110.15779/) (2021)
- [Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.12896/) (2022)
- [Scalability of Message Encoding Techniques for Continuous Communication Learned with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.04844/) (2023)
- [An In-Depth Analysis of Discretization Methods for Communication Learning using Backpropagation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.04938/) (2023)
- [Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents](https://mlsanity.com/paper/2609.33875/) (2026)

## Abstract (arXiv)

Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning (MARL). The credit assignment problem, the non-stationarity of the communication environment and the creation of influenceable agents are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper introduces the novel multi-agent counterfactual communication learning (MACC) method which adapts counterfactual reasoning in order to overcome the credit assignment problem for communicating agents. Secondly, the non-stationarity of the communication environment while learning the communication Q-function is overcome by creating the communication Q-function using the action policy of the other agents and the Q-function of the action environment. Additionally, a social loss function is introduced in order to create influenceable agents which is required to learn a valid communication protocol. Our experiments show that MACC is able to outperform the state-of-the-art baselines in four different scenarios in the Particle environment.