mlsanityData to 14 Oct 2026

Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audio

arXiv 2308.05218 ↗First posted 9 Aug 2023cs.SD cs.LG eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

We propose CONF-TSASR, a non-autoregressive end-to-end time-frequency domain architecture for single-channel target-speaker automatic speech recognition (TS-ASR). The model consists of a TitaNet based speaker embedding module, a Conformer based masking as well as ASR modules. These modules are jointly optimized to transcribe a target-speaker, while ignoring speech from other speakers. For training we use Connectionist Temporal Classification (CTC) loss and introduce a scale-invariant spectrogram reconstruction loss to encourage the model better separate the target-speaker's spectrogram from mixture. We obtain state-of-the-art target-speaker word error rate (TS-WER) on WSJ0-2mix-extr (4.2%). Further, we report for the first time TS-WER on WSJ0-3mix-extr (12.4%), LibriSpeech2Mix (4.2%) and LibriSpeech3Mix (7.6%) datasets, establishing new benchmarks for TS-ASR. The proposed model will be open-sourced through NVIDIA NeMo toolkit.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

18 of 24 references.

References · 18

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  3. 2018SDR - half-baked or well done?cited by 423
  4. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  5. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  6. 2020Attention is All You Need in Speech Separationcited by 185
  7. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 98
  8. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  9. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  10. 2021TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global contextcited by 62
  11. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  12. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
6 more
  1. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  2. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  3. 2020Streaming Multi-speaker ASR with RNN-Tcited by 23
  4. 2021Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chaincited by 12
  5. 2022Streaming Target-Speaker ASR with Neural Transducercited by 9
  6. 2022Semi-supervised Time Domain Target Speaker Extraction with Attentioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 0; 2026: 3.

10 papers.

  1. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  2. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  3. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  4. 2024Alignment-Free Training for Transducer-based Multi-Talker ASRcited by 4
  5. 2024Investigation of Speaker Representation for Target-Speaker Speech Processingcited by 2
  6. 2026Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPOcited by 2
  7. 2023A Glance is Enough: Extract Target Sentence By Looking at A keywordcited by 1
  8. 2024Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokenscited by 1
  9. 2026Xiaomi-CocktailASR-1 Technical Reportcited by 1
  10. 2026CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASRcited by 0