# Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channel

arXiv: https://arxiv.org/abs/2111.02827 · HTML: https://mlsanity.com/paper/2111.02827/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols](https://mlsanity.com/paper/1705.11192/) (2017)
- [Compositionality and Generalization in Emergent Languages](https://mlsanity.com/paper/2004.09124/) (2020)
- [Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input](https://mlsanity.com/paper/1804.03984/) (2018)
- [The Optimal Reward Baseline for Gradient-Based Reinforcement Learning](https://mlsanity.com/paper/1301.2315/) (2013)
- [Biases for Emergent Communication in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1912.05676/) (2019)
- [EGG: a toolkit for research on Emergence of lanGuage in Games](https://mlsanity.com/paper/1907.00852/) (2019)
- [Capacity, Bandwidth, and Compositionality in Emergent Language Learning](https://mlsanity.com/paper/1910.11424/) (2019)
- ["LazImpa": Lazy and Impatient neural agents learn to communicate efficiently](https://mlsanity.com/paper/2010.01878/) (2020)
- [Learning to cooperate: Emergent communication in multi-agent navigation](https://mlsanity.com/paper/2004.01097/) (2020)
- [Emergence of Pragmatics from Referential Game between Theory of Mind Agents](https://mlsanity.com/paper/2001.07752/) (2020)
- [On the Spontaneous Emergence of Discrete and Compositional Signals](https://mlsanity.com/paper/2005.00110/) (2020)
- [Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges](https://mlsanity.com/paper/2002.08878/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [ELCC: the Emergent Language Corpus Collection](https://mlsanity.com/paper/2407.04158/) (2024)
- [XferBench: a Data-Driven Benchmark for Emergent Language](https://mlsanity.com/paper/2407.03456/) (2024)

## Abstract (arXiv)

Multi-agent reinforcement learning has been used as an effective means to study emergent communication between agents, yet little focus has been given to continuous acoustic communication. This would be more akin to human language acquisition; human infants acquire language in large part through continuous signalling with their caregivers. We therefore ask: Are we able to observe emergent language between agents with a continuous communication channel? Our goal is to provide a platform to begin bridging the gap between human and agent communication, allowing us to analyse continuous signals, how they emerge, their characteristics, and how they relate to human language acquisition. We propose a messaging environment where a Speaker agent needs to convey a set of attributes to a Listener over a noisy acoustic channel. Using DQN to train our agents, we show that: (1) unlike the discrete case, the acoustic Speaker learns redundancy to improve Listener coherency, (2) the acoustic Speaker develops more compositional communication protocols which implicitly compensates for transmission errors over a noisy channel, and (3) DQN has significant performance gains and increased compositionality when compared to previous methods optimised using REINFORCE.