# Agents that Listen: High-Throughput Reinforcement Learning with Multiple Sensory Systems

arXiv: https://arxiv.org/abs/2107.02195 · HTML: https://mlsanity.com/paper/2107.02195/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2006.11751/) (2020)
- [ViZDoom Competitions: Playing Doom from Pixels](https://mlsanity.com/paper/1809.03470/) (2018)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [Do Autonomous Agents Benefit from Hearing?](https://mlsanity.com/paper/1905.04192/) (2019)
- ["Did You Hear That?" Learning to Play Video Games from Audio Cues](https://mlsanity.com/paper/1906.04027/) (2019)
- [VECA : A Toolkit for Building Virtual Environments to Train and Test Human-like Agents](https://mlsanity.com/paper/2105.00762/) (2021)

## Built on by (most cited first, 4 of 4)

- [DareFightingICE Competition: A Fighting Game Sound Design and AI Competition](https://mlsanity.com/paper/2203.01556/) (2022)
- [A Deep Reinforcement Learning Blind AI in DareFightingICE](https://mlsanity.com/paper/2205.07444/) (2022)
- [A Deep Reinforcement Learning Approach for Audio-based Navigation and Audio Source Localization in Multi-speaker Environments](https://mlsanity.com/paper/2110.12778/) (2021)
- [Improving Post-Processing of Audio Event Detectors Using Reinforcement Learning](https://mlsanity.com/paper/2208.09201/) (2022)

## Abstract (arXiv)

Humans and other intelligent animals evolved highly sophisticated perception systems that combine multiple sensory modalities. On the other hand, state-of-the-art artificial agents rely mostly on visual inputs or structured low-dimensional observations provided by instrumented environments. Learning to act based on combined visual and auditory inputs is still a new topic of research that has not been explored beyond simple scenarios. To facilitate progress in this area we introduce a new version of VizDoom simulator to create a highly efficient learning environment that provides raw audio observations. We study the performance of different model architectures in a series of tasks that require the agent to recognize sounds and execute instructions given in natural language. Finally, we train our agent to play the full game of Doom and find that it can consistently defeat a traditional vision-based adversary. We are currently in the process of merging the augmented simulator with the main ViZDoom code repository. Video demonstrations and experiment code can be found at https://sites.google.com/view/sound-rl.