# A Deep Reinforcement Learning Approach to Audio-Based Navigation in a Multi-Speaker Environment

arXiv: https://arxiv.org/abs/2105.04488 · HTML: https://mlsanity.com/paper/2105.04488/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Deep Neural Networks for Multiple Speaker Detection and Localization](https://mlsanity.com/paper/1711.11565/) (2017)
- [Towards End-to-End Acoustic Localization using Deep Learning: from Audio Signal to Source Position Coordinates](https://mlsanity.com/paper/1807.11094/) (2018)
- [Deep learning methods in speaker recognition: a review](https://mlsanity.com/paper/1911.06615/) (2019)
- [Do Autonomous Agents Benefit from Hearing?](https://mlsanity.com/paper/1905.04192/) (2019)
- [Multi-Speaker Localization Using Convolutional Neural Network Trained with Noise](https://mlsanity.com/paper/1712.04276/) (2017)

## Built on by (most cited first, 6 of 6)

- [Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning](https://mlsanity.com/paper/2403.05770/) (2024)
- [A Deep Reinforcement Learning Blind AI in DareFightingICE](https://mlsanity.com/paper/2205.07444/) (2022)
- [Audio-Driven Reinforcement Learning for Head-Orientation in Naturalistic Environments](https://mlsanity.com/paper/2409.10048/) (2024)
- [A Deep Reinforcement Learning Approach for Audio-based Navigation and Audio Source Localization in Multi-speaker Environments](https://mlsanity.com/paper/2110.12778/) (2021)
- [Improving Post-Processing of Audio Event Detectors Using Reinforcement Learning](https://mlsanity.com/paper/2208.09201/) (2022)
- [Simulating Human Audiovisual Search Behavior](https://mlsanity.com/paper/2602.02790/) (2026)

## Abstract (arXiv)

In this work we use deep reinforcement learning to create an autonomous agent that can navigate in a two-dimensional space using only raw auditory sensory information from the environment, a problem that has received very little attention in the reinforcement learning literature. Our experiments show that the agent can successfully identify a particular target speaker among a set of N predefined speakers in a room and move itself towards that speaker, while avoiding collision with other speakers or going outside the room boundaries. The agent is shown to be robust to speaker pitch shifting and it can learn to navigate the environment, even when a limited number of training utterances are available for each speaker.