mlsanityData to 7 Oct 2026

Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone

arXiv 2103.16776 ↗First posted 31 Mar 2021eess.AS cs.CL cs.SDcited by 33

Not on the map. Why?

Abstract (arXiv)

Transcribing meetings containing overlapped speech with only a single distant microphone (SDM) has been one of the most challenging problems for automatic speech recognition (ASR). While various approaches have been proposed, all previous studies on the monaural overlapped speech recognition problem were based on either simulation data or small-scale real data. In this paper, we extensively investigate a two-step approach where we first pre-train a serialized output training (SOT)-based multi-talker ASR by using large-scale simulation data and then fine-tune the model with a small amount of real meeting data. Experiments are conducted by utilizing 75 thousand (K) hours of our internal single-talker recording to simulate a total of 900K hours of multi-talker audio segments for supervised pre-training. With fine-tuning on the 70 hours of the AMI-SDM training data, our SOT ASR model achieves a word error rate (WER) of 21.2% for the AMI-SDM evaluation set while automatically counting speakers in each test segment. This result is not only significantly better than the previous state-of-the-art WER of 36.4% with oracle utterance boundary information but also better than a result by a similarly fine-tuned single-talker ASR model applied to beamformed audio.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

24 of 43 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  7. 2015Attention-Based Models for Speech Recognitioncited by 400
  8. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  10. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  11. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  12. 2020Continuous speech separation: dataset and analysiscited by 73
12 more
  1. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  2. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  3. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  4. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 34
  5. 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
  6. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  7. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
  8. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  9. 2019Advances in Online Audio-Visual Meeting Transcriptioncited by 23
  10. 2020Streaming Multi-speaker ASR with RNN-Tcited by 22
  11. 2020Transfer Learning Approaches for Streaming End-to-End Speech Recognition Systemcited by 10
  12. 2021Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordingscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 6; 2024: 5; 2025: 2; 2026: 4.

33 papers.

Sort:

By area:

  1. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  2. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  3. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  4. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  5. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  6. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  7. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audiocited by 10
  8. 2021Multi-turn RNN-T for streaming recognition of multi-party speechcited by 10
  9. 2022Simulating realistic speech overlaps improves multi-talker ASRcited by 10
  10. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  11. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
  12. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
21 more
  1. 2022Speech separation with large-scale self-supervised learningcited by 5
  2. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  3. 2024Serialized Output Training by Learned Dominancecited by 5
  4. 2026TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Groundingcited by 5
  5. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  6. 2021Are E2E ASR models ready for an industrial usage?cited by 3
  7. 2022The RoyalFlush System of Speech Recognition for M2MeT Challengecited by 3
  8. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  9. 2023End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysiscited by 3
  10. 2024Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applicationscited by 3
  11. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 3
  12. 2022Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Expertscited by 2
  13. 2023Using Kaldi for Automatic Speech Recognition of Conversational Austrian Germancited by 2
  14. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognitioncited by 2
  15. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  16. 2021Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognitioncited by 1
  17. 2022Enhancing Speech Recognition Decoding via Layer Aggregationcited by 1
  18. 2023Factual Consistency Oriented Speech Recognitioncited by 1
  19. 2023Speech Corpora Divergence Based Unsupervised Data Selection for ASRcited by 0
  20. 2026Modeling Overlapped Speech with Shufflescited by 0
  21. 2026Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correctioncited by 0