# End-to-end multi-channel speaker extraction and binaural speech synthesis

arXiv: https://arxiv.org/abs/2410.05739 · HTML: https://mlsanity.com/paper/2410.05739/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [2.5D Visual Sound](https://mlsanity.com/paper/1812.04204/) (2018)
- [Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention](https://mlsanity.com/paper/2111.08046/) (2021)
- [Embedding and Beamforming: All-neural Causal Beamformer for Multichannel Speech Enhancement](https://mlsanity.com/paper/2109.00265/) (2021)
- [3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications](https://mlsanity.com/paper/2312.10979/) (2023)
- [TaylorBeamformer: Learning All-Neural Beamformer for Multi-Channel Speech Enhancement from Taylor's Approximation Theory](https://mlsanity.com/paper/2203.07195/) (2022)
- [Model-matching Principle Applied to the Design of an Array-based All-neural Binaural Rendering System for Audio Telepresence](https://mlsanity.com/paper/2210.11123/) (2022)
- [DopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect](https://mlsanity.com/paper/2212.07000/) (2022)

## Abstract (arXiv)

Speech clarity and spatial audio immersion are the two most critical factors in enhancing remote conferencing experiences. Existing methods are often limited: either due to the lack of spatial information when using only one microphone, or because their performance is highly dependent on the accuracy of direction-of-arrival estimation when using microphone array. To overcome this issue, we introduce an end-to-end deep learning framework that has the capacity of mapping multi-channel noisy and reverberant signals to clean and spatialized binaural speech directly. This framework unifies source extraction, noise suppression, and binaural rendering into one network. In this framework, a novel magnitude-weighted interaural level difference loss function is proposed that aims to improve the accuracy of spatial rendering. Extensive evaluations show that our method outperforms established baselines in terms of both speech quality and spatial fidelity.