# Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation

arXiv: https://arxiv.org/abs/2204.03232 · HTML: https://mlsanity.com/paper/2204.03232/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Continuous Speech Separation with Conformer](https://mlsanity.com/paper/2008.05773/) (2020)
- [Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis](https://mlsanity.com/paper/2011.02014/) (2020)
- [Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networks](https://mlsanity.com/paper/1810.03655/) (2018)
- [Advances in Online Audio-Visual Meeting Transcription](https://mlsanity.com/paper/1912.04979/) (2019)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.11458/) (2020)
- [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021)
- [Self-supervised Learning for Speech Enhancement](https://mlsanity.com/paper/2006.10388/) (2020)
- [Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement](https://mlsanity.com/paper/2106.02896/) (2021)
- [Investigation of Practical Aspects of Single Channel Speech Separation for ASR](https://mlsanity.com/paper/2107.01922/) (2021)
- [Exploring End-to-End Multi-channel ASR with Bias Information for Meeting Transcription](https://mlsanity.com/paper/2011.03110/) (2020)
- [Incorporating Real-world Noisy Speech in Neural-network-based Speech Enhancement Systems](https://mlsanity.com/paper/2109.05172/) (2021)
- [An Investigation of End-to-End Multichannel Speech Recognition for Reverberant and Mismatch Conditions](https://mlsanity.com/paper/1904.09049/) (2019)
- [The CUHK-TENCENT speaker diarization system for the ICASSP 2022 multi-channel multi-party meeting transcription challenge](https://mlsanity.com/paper/2202.01986/) (2022)
- [All-neural beamformer for continuous speech separation](https://mlsanity.com/paper/2110.06428/) (2021)

## Built on by (most cited first, 4 of 4)

- [VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition](https://mlsanity.com/paper/2209.04974/) (2022)
- [Speech separation with large-scale self-supervised learning](https://mlsanity.com/paper/2211.05172/) (2022)
- [Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts](https://mlsanity.com/paper/2211.06493/) (2022)
- [Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization](https://mlsanity.com/paper/2210.03459/) (2022)

## Abstract (arXiv)

Existing multi-channel continuous speech separation (CSS) models are heavily dependent on supervised data - either simulated data which causes data mismatch between the training and real-data testing, or the real transcribed overlapping data, which is difficult to be acquired, hindering further improvements in the conversational/meeting transcription tasks. In this paper, we propose a three-stage training scheme for the CSS model that can leverage both supervised data and extra large-scale unsupervised real-world conversational data. The scheme consists of two conventional training approaches -- pre-training using simulated data and ASR-loss-based training using transcribed data -- and a novel continuous semi-supervised training between the two, in which the CSS model is further trained by using real data based on the teacher-student learning framework. We apply this scheme to an array-geometry-agnostic CSS model, which can use the multi-channel data collected from any microphone array. Large-scale meeting transcription experiments are carried out on both Microsoft internal meeting data and the AMI meeting corpus. The steady improvement by each training stage has been observed, showing the effect of the proposed method that enables leveraging real conversational data for CSS model training.