# A Deep Reinforcement Learning Approach for Audio-based Navigation and Audio Source Localization in Multi-speaker Environments

arXiv: https://arxiv.org/abs/2110.12778 · HTML: https://mlsanity.com/paper/2110.12778/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Deep Neural Networks for Multiple Speaker Detection and Localization](https://mlsanity.com/paper/1711.11565/) (2017)
- [Towards End-to-End Acoustic Localization using Deep Learning: from Audio Signal to Source Position Coordinates](https://mlsanity.com/paper/1807.11094/) (2018)
- [Deep learning methods in speaker recognition: a review](https://mlsanity.com/paper/1911.06615/) (2019)
- [A Deep Reinforcement Learning Approach to Audio-Based Navigation in a Multi-Speaker Environment](https://mlsanity.com/paper/2105.04488/) (2021)
- [Do Autonomous Agents Benefit from Hearing?](https://mlsanity.com/paper/1905.04192/) (2019)
- [Multi-Speaker Localization Using Convolutional Neural Network Trained with Noise](https://mlsanity.com/paper/1712.04276/) (2017)
- [Agents that Listen: High-Throughput Reinforcement Learning with Multiple Sensory Systems](https://mlsanity.com/paper/2107.02195/) (2021)

## Built on by (most cited first, 1 of 1)

- [Audio-Driven Reinforcement Learning for Head-Orientation in Naturalistic Environments](https://mlsanity.com/paper/2409.10048/) (2024)

## Abstract (arXiv)

In this work we apply deep reinforcement learning to the problems of navigating a three-dimensional environment and inferring the locations of human speaker audio sources within, in the case where the only available information is the raw sound from the environment, as a simulated human listener placed in the environment would hear it. For this purpose we create two virtual environments using the Unity game engine, one presenting an audio-based navigation problem and one presenting an audio source localization problem. We also create an autonomous agent based on PPO online reinforcement learning algorithm and attempt to train it to solve these environments. Our experiments show that our agent achieves adequate performance and generalization ability in both environments, measured by quantitative metrics, even when a limited amount of training data are available or the environment parameters shift in ways not encountered during training. We also show that a degree of agent knowledge transfer is possible between the environments.