mlsanityData to 19 Oct 2026

Cascaded encoders for fine-tuning ASR models on overlapped speech

arXiv 2306.16398 ↗First posted 28 Jun 2023cs.SD eess.AScited by 1

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

Multi-talker speech recognition (MT-ASR) has been shown to improve ASR performance on speech containing overlapping utterances from more than one speaker. Multi-talker models have typically been trained from scratch using simulated or actual overlapping speech datasets. On the other hand, the trend in ASR has been to train foundation models using massive datasets collected from a wide variety of task domains. Given the scale of these models and their ability to generalize well across a variety of domains, it makes sense to consider scenarios where a foundation model is augmented with multi-talker capability. This paper presents an MT-ASR model formed by combining a well-trained foundation model with a multi-talker mask model in a cascaded RNN-T encoder configuration. Experimental results show that the cascade configuration provides improved WER on overlapping speech utterances with respect to a baseline multi-talker model without sacrificing performance achievable by the foundation model on non-overlapping utterances.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 22 of its 28 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

22 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Multi-turn RNN-T for streaming recognition of multi-party speech19 Dec
  2. 2021Continuous Streaming Multi-Talker ASR with Dual-path Transducers17 Sep

Other references · 20

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 2,293
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,410
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 286
  6. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 247
  7. 2020Continuous speech separation: dataset and analysiscited by 124
  8. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 97
  9. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 80
  10. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 67
  11. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 64
  12. 2018Large-Scale Visual Speech Recognitioncited by 58
8 more
  1. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 58
  2. 2020Streaming end-to-end multi-talker speech recognitioncited by 41
  3. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 37
  4. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 34
  5. 2021Graph-PIT: Generalized permutation invariant training for continuous separation of arbitrary numbers of speakerscited by 14
  6. 2020FEARLESS STEPS Challenge (FS-2): Supervised Learning with Massive Naturalistic Apollo Datacited by 7
  7. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 6
  8. 2022End-to-end multi-talker audio-visual ASR using an active speaker attention modulecited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2022Simulating realistic speech overlaps improves multi-talker ASR
  2. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone
  3. 2024Advancing Multi-talker ASR Performance with Large Language Models
  4. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One
  5. 2024META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASR
  6. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkers
  7. 2022A Unified Cascaded Encoder ASR Model for Dynamic Model Sizes
  8. 2026Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing