# An End-to-end Architecture of Online Multi-channel Speech Separation

arXiv: https://arxiv.org/abs/2009.03141 · HTML: https://mlsanity.com/paper/2009.03141/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [Implicit Neural Spatial Filtering for Multichannel Source Separation in the Waveform Domain](https://mlsanity.com/paper/2206.15423/) (2022)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR](https://mlsanity.com/paper/2305.13716/) (2023)
- [Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separation](https://mlsanity.com/paper/2505.13094/) (2025)
- [Online Self-Attentive Gated RNNs for Real-Time Speaker Separation](https://mlsanity.com/paper/2106.13493/) (2021)

## Abstract (arXiv)

Multi-speaker speech recognition has been one of the keychallenges in conversation transcription as it breaks the singleactive speaker assumption employed by most state-of-the-artspeech recognition systems. Speech separation is consideredas a remedy to this problem. Previously, we introduced a sys-tem, calledunmixing,fixed-beamformerandextraction(UFE),that was shown to be effective in addressing the speech over-lap problem in conversation transcription. With UFE, an inputmixed signal is processed by fixed beamformers, followed by aneural network post filtering. Although promising results wereobtained, the system contains multiple individually developedmodules, leading potentially sub-optimum performance. In thiswork, we introduce an end-to-end modeling version of UFE. Toenable gradient propagation all the way, an attentional selectionmodule is proposed, where an attentional weight is learnt foreach beamformer and spatial feature sampled over space. Ex-perimental results show that the proposed system achieves com-parable performance in an offline evaluation with the originalseparate processing-based pipeline, while producing remark-able improvements in an online evaluation.