# Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

arXiv: https://arxiv.org/abs/1602.02672 · HTML: https://mlsanity.com/paper/1602.02672/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2016.

## Built on by (most cited first, 21 of 21)

- [Learning to Ground Multi-Agent Communication with Autoencoders](https://mlsanity.com/paper/2110.15349/) (2021)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives](https://mlsanity.com/paper/2503.13415/) (2025)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [Generative Emergent Communication: Large Language Model is a Collective World Model](https://mlsanity.com/paper/2501.00226/) (2024)
- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [Emergence of Theory of Mind Collaboration in Multiagent Systems](https://mlsanity.com/paper/2110.00121/) (2021)
- [Approximate information state based convergence analysis of recurrent Q-learning](https://mlsanity.com/paper/2306.05991/) (2023)
- [Visual Spatial Attention and Proprioceptive Data-Driven Reinforcement Learning for Robust Peg-in-Hole Task Under Variable Conditions](https://mlsanity.com/paper/2312.16438/) (2023)
- [Behavior-Inspired Neural Networks for Relational Inference](https://mlsanity.com/paper/2406.14746/) (2024)
- [Deep Reinforcement Learning for Demand Driven Services in Logistics and Transportation Systems: A Survey](https://mlsanity.com/paper/2108.04462/) (2021)
- [CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learning](https://mlsanity.com/paper/2306.11128/) (2023)
- [Resilient robot teams: a review integrating decentralised control, change-detection, and learning](https://mlsanity.com/paper/2204.10063/) (2022)
- [Model-free Motion Planning of Autonomous Agents for Complex Tasks in Partially Observable Environments](https://mlsanity.com/paper/2305.00561/) (2023)
- [GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactions](https://mlsanity.com/paper/2609.01491/) (2026)
- [Socially Supervised Representation Learning: the Role of Subjectivity in Learning Efficient Representations](https://mlsanity.com/paper/2109.09390/) (2021)
- [A Decentralized Communication Framework based on Dual-Level Recurrence for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.10612/) (2022)
- [Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2505.11153/) (2025)
- [Adaptive Event-Triggered Policy Gradient for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.20338/) (2025)
- [Provably Optimal Learning Algorithms for Assistance Games](https://mlsanity.com/paper/2607.08012/) (2026)

## Abstract (arXiv)

We propose deep distributed recurrent Q-networks (DDRQN), which enable teams of agents to learn to solve communication-based coordination tasks. In these tasks, the agents are not given any pre-designed communication protocol. Therefore, in order to successfully communicate, they must first automatically develop and agree upon their own communication protocol. We present empirical results on two multi-agent learning problems based on well-known riddles, demonstrating that DDRQN can successfully solve such tasks and discover elegant communication protocols to do so. To our knowledge, this is the first time deep reinforcement learning has succeeded in learning communication protocols. In addition, we present ablation experiments that confirm that each of the main components of the DDRQN architecture are critical to its success.