mlsanityData to 7 Oct 2026

A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One

arXiv 2302.09908 ↗First posted 20 Feb 2023cs.SD cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Although automatic speech recognition (ASR) can perform well in common non-overlapping environments, sustaining performance in multi-talker overlapping speech recognition remains challenging. Recent research revealed that ASR model's encoder captures different levels of information with different layers -- the lower layers tend to have more acoustic information, and the upper layers more linguistic. This inspires us to develop a Sidecar separator to empower a well-trained ASR model for multi-talker scenarios by separating the mixed speech embedding between two suitable layers. We experimented with a wav2vec 2.0-based ASR model with a Sidecar mounted. By freezing the parameters of the original model and training only the Sidecar (8.7 M, 8.4% of all parameters), the proposed approach outperforms the previous state-of-the-art by a large margin for the 2-speaker mixed LibriMix dataset, reaching a word error rate (WER) of 10.36%; and obtains comparable results (7.56%) for LibriSpeechMix dataset when limited training.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

23 of 32 references.

References · 23

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2015Listen, Attend and Spellcited by 542
  5. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  6. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  7. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  8. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  9. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  10. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  11. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  12. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
11 more
  1. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  2. 2020Speaker Recognition Based on Deep Learning: An Overviewcited by 59
  3. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  4. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  5. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  6. 2022Self-supervised models of audio effectively explain human cortical responses to speechcited by 37
  7. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  8. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  9. 2019What does a network layer hear? Analyzing hidden representations of end-to-end ASR through speech synthesiscited by 13
  10. 2021Streaming Multi-talker Speech Recognition with Joint Speaker Identificationcited by 12
  11. 2021Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chaincited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 2; 2026: 1.

13 papers.

Sort:

  1. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  2. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  3. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  4. 2024Cross-Speaker Encoding Network for Multi-Talker Speech Recognitioncited by 9
  5. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  6. 2024Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognitioncited by 7
  7. 2024Serialized Output Training by Learned Dominancecited by 5
  8. 2025Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognitioncited by 5
  9. 2024Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTCcited by 4
  10. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  11. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 3
  12. 2023A Glance is Enough: Extract Target Sentence By Looking at A keywordcited by 1
1 more
  1. 2026Asymmetric Classifier-Free Guidance for Target-Speaker ASRcited by 0