mlsanityData to 7 Oct 2026

Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition

arXiv 2409.00815 ↗First posted 1 Sep 2024cs.SD cs.AI eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Serialized output training (SOT) attracts increasing attention due to its convenience and flexibility for multi-speaker automatic speech recognition (ASR). However, it is not easy to train with attention loss only. In this paper, we propose the overlapped encoding separation (EncSep) to fully utilize the benefits of the connectionist temporal classification (CTC) and attention hybrid loss. This additional separator is inserted after the encoder to extract the multi-speaker information with CTC losses. Furthermore, we propose the serialized speech information guidance SOT (GEncSep) to further utilize the separated encodings. The separated streams are concatenated to provide single-speaker information to guide attention during decoding. The experimental results on LibriMix show that the single-speaker encoding can be separated from the overlapped encoding. The CTC loss helps to improve the encoder representation under complex scenarios. GEncSep further improved performance.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

29 of 49 references.

References · 29

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  3. 2015Listen, Attend and Spellcited by 542
  4. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  5. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  6. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  7. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  8. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  9. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  10. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  11. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  12. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
17 more
  1. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  2. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  3. 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
  4. 2020Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASRcited by 15
  5. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  6. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  7. 2021Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chaincited by 12
  8. 2023Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognitioncited by 11
  9. 2024MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correctioncited by 8
  10. 2023Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speechcited by 6
  11. 2023How does end-to-end speech recognition training impact speech enhancement artifacts?cited by 6
  12. 2023Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognitioncited by 5
  13. 2024Exploration of Adapter for Noise Robust Automatic Speech Recognitioncited by 4
  14. 2023Time-domain Speech Enhancement Assisted by Multi-resolution Frequency Encoder and Decodercited by 3
  15. 2024Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Frameworkcited by 3
  16. 2023Speaker Mask Transformer for Multi-talker Overlapped Speech Recognitioncited by 2
  17. 2024Iterative Prototype Refinement for Ambiguous Speech Emotion Recognitioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 7
  2. 2025Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognitioncited by 5
  3. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  4. 2025Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognitioncited by 3
  5. 2025CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Modelscited by 1
  6. 2026Modeling Overlapped Speech with Shufflescited by 0
  7. 2026Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognitioncited by 0