mlsanityData to 7 Oct 2026

Streaming Multi-talker Speech Recognition with Joint Speaker Identification

arXiv 2104.02109 ↗First posted 5 Apr 2021cs.SD cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

In multi-talker scenarios such as meetings and conversations, speech processing systems are usually required to transcribe the audio as well as identify the speakers for downstream applications. Since overlapped speech is common in this case, conventional approaches usually address this problem in a cascaded fashion that involves speech separation, speech recognition and speaker identification that are trained independently. In this paper, we propose Streaming Unmixing, Recognition and Identification Transducer (SURIT) -- a new framework that deals with this problem in an end-to-end streaming fashion. SURIT employs the recurrent neural network transducer (RNN-T) as the backbone for both speech recognition and speaker identification. We validate our idea on the LibrispeechMix dataset -- a multi-talker dataset derived from Librispeech, and present encouraging results.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

26 of 42 references.

References · 26

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  5. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  6. 2015Listen, Attend and Spellcited by 542
  7. 2015Attention-Based Models for Speech Recognitioncited by 400
  8. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  9. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  10. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  11. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  12. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
14 more
  1. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 86
  2. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 66
  3. 2015End-to-End Text-Dependent Speaker Verificationcited by 49
  4. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  5. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  6. 2018Fully Supervised Speaker Diarizationcited by 46
  7. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  8. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  9. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  10. 2018Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networkscited by 23
  11. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  12. 2020Streaming Multi-speaker ASR with RNN-Tcited by 22
  13. 2020Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASRcited by 11
  14. 2016Segmental Recurrent Neural Networks for End-to-end Speech Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 4; 2024: 2; 2025: 0; 2026: 1.

12 papers.

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  3. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  4. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  5. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  6. 2021Multi-turn RNN-T for streaming recognition of multi-party speechcited by 10
  7. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  8. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  9. 2023Mixture Encoder for Joint Speech Separation and Recognitioncited by 4
  10. 2024Alignment-Free Training for Transducer-based Multi-Talker ASRcited by 4
  11. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  12. 2023End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysiscited by 3