# MA-Dreamer: Coordination and communication through shared imagination

arXiv: https://arxiv.org/abs/2204.04687 · HTML: https://mlsanity.com/paper/2204.04687/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols](https://mlsanity.com/paper/1705.11192/) (2017)
- [Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input](https://mlsanity.com/paper/1804.03984/) (2018)
- [Emergent Communication through Negotiation](https://mlsanity.com/paper/1804.03980/) (2018)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)
- [Deep Reinforcement Learning For Sequence to Sequence Models](https://mlsanity.com/paper/1805.09461/) (2018)
- [Learning to cooperate: Emergent communication in multi-agent navigation](https://mlsanity.com/paper/2004.01097/) (2020)
- [Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence](https://mlsanity.com/paper/1611.03218/) (2016)

## Built on by (most cited first, 6 of 6)

- [Generative Emergent Communication: Large Language Model is a Collective World Model](https://mlsanity.com/paper/2501.00226/) (2024)
- [Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.17373/) (2024)
- [Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.07463/) (2026)
- [Learning Multi-agent Multi-machine Tending by Mobile Robots](https://mlsanity.com/paper/2408.16875/) (2024)
- [Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation](https://mlsanity.com/paper/2508.05535/) (2025)
- [Dreaming Of Others: Latent Teammate Modeling In World Models For Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.31361/) (2026)

## Abstract (arXiv)

Multi-agent RL is rendered difficult due to the non-stationary nature of environment perceived by individual agents. Theoretically sound methods using the REINFORCE estimator are impeded by its high-variance, whereas value-function based methods are affected by issues stemming from their ad-hoc handling of situations like inter-agent communication. Methods like MADDPG are further constrained due to their requirement of centralized critics etc. In order to address these issues, we present MA-Dreamer, a model-based method that uses both agent-centric and global differentiable models of the environment in order to train decentralized agents' policies and critics using model-rollouts a.k.a `imagination'. Since only the model-training is done off-policy, inter-agent communication/coordination and `language emergence' can be handled in a straight-forward manner. We compare the performance of MA-Dreamer with other methods on two soccer-based games. Our experiments show that in long-term speaker-listener tasks and in cooperative games with strong partial-observability, MA-Dreamer finds a solution that makes effective use of coordination, whereas competing methods obtain marginal scores and fail outright, respectively. By effectively achieving coordination and communication under more relaxed and general conditions, out method opens the door to the study of more complex problems and population-based training.