# Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition

arXiv: https://arxiv.org/abs/2102.07955 · HTML: https://mlsanity.com/paper/2102.07955/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2021.

## Built on by (most cited first, 14 of 14)

- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis](https://mlsanity.com/paper/2311.01052/) (2023)
- [BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR](https://mlsanity.com/paper/2305.13716/) (2023)
- [MIMO-DoAnet: Multi-channel Input and Multiple Outputs DoA Network with Unknown Number of Sound Sources](https://mlsanity.com/paper/2207.07307/) (2022)
- [Multi-Microphone Speaker Separation by Spatial Regions](https://mlsanity.com/paper/2303.07143/) (2023)
- [Position tracking of a varying number of sound sources with sliding permutation invariant training](https://mlsanity.com/paper/2210.14536/) (2022)
- [Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers](https://mlsanity.com/paper/2506.19875/) (2025)
- [Learning Robust Spatial Representations from Binaural Audio through Feature Distillation](https://mlsanity.com/paper/2508.20914/) (2025)
- [Iterative Sound Source Localization for Unknown Number of Sources](https://mlsanity.com/paper/2206.12273/) (2022)
- [A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods](https://mlsanity.com/paper/2507.01143/) (2025)
- [Sound field decomposition based on two-stage neural networks](https://mlsanity.com/paper/2309.06661/) (2023)
- [Feature Aggregation in Joint Sound Classification and Localization Neural Networks](https://mlsanity.com/paper/2310.19063/) (2023)
- [Audio Simulation for Sound Source Localization in Virtual Evironment](https://mlsanity.com/paper/2404.01611/) (2024)
- [Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings](https://mlsanity.com/paper/2508.14115/) (2025)

## Abstract (arXiv)

Multi-source localization is an important and challenging technique for multi-talker conversation analysis. This paper proposes a novel supervised learning method using deep neural networks to estimate the direction of arrival (DOA) of all the speakers simultaneously from the audio mixture. At the heart of the proposal is a source splitting mechanism that creates source-specific intermediate representations inside the network. This allows our model to give source-specific posteriors as the output unlike the traditional multi-label classification approach. Existing deep learning methods perform a frame level prediction, whereas our approach performs an utterance level prediction by incorporating temporal selection and averaging inside the network to avoid post-processing. We also experiment with various loss functions and show that a variant of earth mover distance (EMD) is very effective in classifying DOA at a very high resolution by modeling inter-class relationships. In addition to using the prediction error as a metric for evaluating our localization model, we also establish its potency as a frontend with automatic speech recognition (ASR) as the downstream task. We convert the estimated DOAs into a feature suitable for ASR and pass it as an additional input feature to a strong multi-channel and multi-talker speech recognition baseline. This added input feature drastically improves the ASR performance and gives a word error rate (WER) of 6.3% on the evaluation data of our simulated noisy two speaker mixtures, while the baseline which doesn't use explicit localization input has a WER of 11.5%. We also perform ASR evaluation on real recordings with the overlapped set of the MC-WSJ-AV corpus in addition to simulated mixtures.