# Interpretable agent communication from scratch (with a generic visual processor emerging on the side)

arXiv: https://arxiv.org/abs/2106.04258 · HTML: https://mlsanity.com/paper/2106.04258/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [Multi-Agent Cooperation and the Emergence of (Natural) Language](https://mlsanity.com/paper/1612.07182/) (2016)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [Discrete Variational Autoencoders](https://mlsanity.com/paper/1609.02200/) (2016)
- [Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols](https://mlsanity.com/paper/1705.11192/) (2017)
- [Compositionality and Generalization in Emergent Languages](https://mlsanity.com/paper/2004.09124/) (2020)

## Built on by (most cited first, 25 of 25)

- [Emergent Communication: Generalization and Overfitting in Lewis Games](https://mlsanity.com/paper/2209.15342/) (2022)
- [Simplicial Embeddings in Self-Supervised Learning and Downstream Classification](https://mlsanity.com/paper/2204.00616/) (2022)
- [Linking Emergent and Natural Languages via Corpus Transfer](https://mlsanity.com/paper/2203.13344/) (2022)
- [Iterated Learning Improves Compositionality in Large Vision-Language Models](https://mlsanity.com/paper/2404.02145/) (2024)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segments](https://mlsanity.com/paper/2311.04453/) (2023)
- [Referential communication in heterogeneous communities of pre-trained visual deep networks](https://mlsanity.com/paper/2302.08913/) (2023)
- [Learning Multi-Agent Communication with Contrastive Learning](https://mlsanity.com/paper/2307.01403/) (2023)
- [Emergence of hierarchical reference systems in multi-agent communication](https://mlsanity.com/paper/2203.13176/) (2022)
- [Emergent Quantized Communication](https://mlsanity.com/paper/2211.02412/) (2022)
- [The Curious Case of Representational Alignment: Unravelling Visio-Linguistic Tasks in Emergent Communication](https://mlsanity.com/paper/2407.17960/) (2024)
- [Language Evolution with Deep Learning](https://mlsanity.com/paper/2403.11958/) (2024)
- [Learning Multi-Object Positional Relationships via Emergent Communication](https://mlsanity.com/paper/2302.08084/) (2023)
- [Speaking Your Language: Spatial Relationships in Interpretable Emergent Communication](https://mlsanity.com/paper/2406.07277/) (2024)
- [Meta-Referential Games to Learn Compositional Learning Behaviours](https://mlsanity.com/paper/2207.08012/) (2022)
- [ELCC: the Emergent Language Corpus Collection](https://mlsanity.com/paper/2407.04158/) (2024)
- [Implicit Repair with Reinforcement Learning in Emergent Communication](https://mlsanity.com/paper/2502.12624/) (2025)
- [Investigating the Development of Task-Oriented Communication in Vision-Language Models](https://mlsanity.com/paper/2601.20641/) (2026)
- [A Compressive-Expressive Communication Framework for Compositional Representations](https://mlsanity.com/paper/2501.19182/) (2025)
- [CtD: Composition through Decomposition in Emergent Communication](https://mlsanity.com/paper/2601.10169/) (2026)
- [ETHER: Aligning Emergent Communication for Hindsight Experience Replay](https://mlsanity.com/paper/2307.15494/) (2023)
- [Emergent Communication in Continuous Worlds: Self-Organisation of Conceptually Grounded Vocabularies at Scale](https://mlsanity.com/paper/2401.08461/) (2024)
- [Cognitively-Inspired Emergent Communication via Knowledge Graphs for Assisting the Visually Impaired](https://mlsanity.com/paper/2505.22087/) (2025)
- [Natural, Artificial, and Human Intelligences](https://mlsanity.com/paper/2506.02183/) (2025)

## Abstract (arXiv)

As deep networks begin to be deployed as autonomous agents, the issue of how they can communicate with each other becomes important. Here, we train two deep nets from scratch to perform realistic referent identification through unsupervised emergent communication. We show that the largely interpretable emergent protocol allows the nets to successfully communicate even about object types they did not see at training time. The visual representations induced as a by-product of our training regime, moreover, show comparable quality, when re-used as generic visual features, to a recent self-supervised learning model. Our results provide concrete evidence of the viability of (interpretable) emergent deep net communication in a more realistic scenario than previously considered, as well as establishing an intriguing link between this field and self-supervised visual learning.