# Neural Blind Source Separation and Diarization for Distant Speech Recognition

arXiv: https://arxiv.org/abs/2406.08396 · HTML: https://mlsanity.com/paper/2406.08396/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [A Recurrent Variational Autoencoder for Speech Enhancement](https://mlsanity.com/paper/1910.10942/) (2019)
- [End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation](https://mlsanity.com/paper/1910.14104/) (2019)
- [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023)
- [Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR](https://mlsanity.com/paper/2006.02786/) (2020)
- [Resource-Efficient Separation Transformer](https://mlsanity.com/paper/2206.09507/) (2022)
- [EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers](https://mlsanity.com/paper/2203.17068/) (2022)
- [GPU-accelerated Guided Source Separation for Meeting Transcription](https://mlsanity.com/paper/2212.05271/) (2022)
- [Unsupervised training of neural mask-based beamforming](https://mlsanity.com/paper/1904.01578/) (2019)
- [VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition](https://mlsanity.com/paper/2209.04974/) (2022)
- [Surrogate Source Model Learning for Determined Source Separation](https://mlsanity.com/paper/2011.05540/) (2020)
- [Unsupervised Speech Enhancement Based on Multichannel NMF-Informed Beamforming for Noise-Robust Automatic Speech Recognition](https://mlsanity.com/paper/1903.09341/) (2019)
- [UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures](https://mlsanity.com/paper/2305.20054/) (2023)
- [The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge](https://mlsanity.com/paper/2308.14638/) (2023)
- [FastMVAE2: On improving and accelerating the fast variational autoencoder-based source separation algorithm for determined mixtures](https://mlsanity.com/paper/2109.13496/) (2021)
- [Neural Fast Full-Rank Spatial Covariance Analysis for Blind Source Separation](https://mlsanity.com/paper/2306.10240/) (2023)

## Built on by (most cited first, 6 of 6)

- [Towards AI-Driven Policing: Interdisciplinary Knowledge Discovery from Police Body-Worn Camera Footage](https://mlsanity.com/paper/2504.20007/) (2025)
- [BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition](https://mlsanity.com/paper/2505.00059/) (2025)
- [Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder](https://mlsanity.com/paper/2508.20474/) (2025)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [CabinSep: IR-Augmented Mask-Based MVDR for Real-Time In-Car Speech Separation with Distributed Heterogeneous Arrays](https://mlsanity.com/paper/2509.01399/) (2025)
- [Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model](https://mlsanity.com/paper/2609.12154/) (2026)

## Abstract (arXiv)

This paper presents a neural method for distant speech recognition (DSR) that jointly separates and diarizes speech mixtures without supervision by isolated signals. A standard separation method for multi-talker DSR is a statistical multichannel method called guided source separation (GSS). While GSS does not require signal-level supervision, it relies on speaker diarization results to handle unknown numbers of active speakers. To overcome this limitation, we introduce and train a neural inference model in a weakly-supervised manner, employing the objective function of a statistical separation method. This training requires only multichannel mixtures and their temporal annotations of speaker activities. In contrast to GSS, the trained model can jointly separate and diarize speech mixtures without any auxiliary information. The experiments with the AMI corpus show that our method outperforms GSS with oracle diarization results regarding word error rates. The code is available online.