mlsanityData to 13 Oct 2026

Adapting Multi-Lingual ASR Models for Handling Multiple Talkers

arXiv 2305.18747 ↗First posted 30 May 2023eess.AS cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages. However, it remains a challenge for these models to recognize overlapped speech, which is often seen in meeting conversations. We propose an approach to adapt USMs for multi-talker ASR. We first develop an enhanced version of serialized output training to jointly perform multi-talker ASR and utterance timestamp prediction. That is, we predict the ASR hypotheses for all speakers, count the speakers, and estimate the utterance timestamps at the same time. We further introduce a lightweight adapter module to maintain the multilingual property of the USMs even when we perform the adaptation with only a single language. Experimental results obtained using the AMI and AliMeeting corpora show that our proposed approach effectively transfers the USMs to a strong multilingual multi-talker ASR model with timestamp prediction capability.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,625
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 657
  9. 2015Listen, Attend and Spellcited by 542
  10. 2015Attention-Based Models for Speech Recognitioncited by 400
  11. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 190
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
20 more
  1. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 131
  2. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  3. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  4. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  5. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  6. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 72
  7. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 68
  8. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
  9. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  10. 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
  11. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  12. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  13. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  14. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  15. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  16. 2017Single-Channel Multi-talker Speech Recognition with Permutation Invariant Trainingcited by 16
  17. 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
  18. 2022Exploring WavLM on Speech Enhancementcited by 8
  19. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  20. 2021Dual-Path Modeling for Long Recording Speech Separation in Meetingscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 4; 2026: 1.

11 papers.

  1. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  2. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  3. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  4. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  5. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
  6. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  7. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 5
  8. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  9. 2025PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systemscited by 2
  10. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognitioncited by 2
  11. 2026SLM-SS: Speech Language Model for Generative Speech Separationcited by 0