mlsanityData to 7 Oct 2026

End-to-End Speaker-Attributed ASR with Transformer

arXiv 2104.02128 ↗First posted 5 Apr 2021eess.AS cs.CL cs.SDcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper presents our recent effort on end-to-end speaker-attributed automatic speech recognition, which jointly performs speaker counting, speech recognition and speaker identification for monaural multi-talker audio. Firstly, we thoroughly update the model architecture that was previously designed based on a long short-term memory (LSTM)-based attention encoder decoder by applying transformer architectures. Secondly, we propose a speaker deduplication mechanism to reduce speaker identification errors in highly overlapped regions. Experimental results on the LibriSpeechMix dataset shows that the transformer-based architecture is especially good at counting the speakers and that the proposed model reduces the speaker-attributed word error rate by 47% over the LSTM-based baseline. Furthermore, for the LibriCSS dataset, which consists of real recordings of overlapped speech, the proposed model achieves concatenated minimum-permutation word error rates of 11.9% and 16.3% with and without target speaker profiles, respectively, both of which are the state-of-the-art results for LibriCSS with the monaural setting.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

26 of 36 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  10. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  11. 2020Continuous speech separation: dataset and analysiscited by 73
  12. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
14 more
  1. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  2. 2020Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengapcited by 35
  3. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 34
  4. 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
  5. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  6. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  7. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
  8. 2019Advances in Online Audio-Visual Meeting Transcriptioncited by 23
  9. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  10. 2020Streaming Multi-speaker ASR with RNN-Tcited by 22
  11. 2018Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinesecited by 19
  12. 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
  13. 2019Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Modelscited by 13
  14. 2020Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASRcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 5; 2024: 12; 2025: 6; 2026: 3.

35 papers.

Sort:

By area:

  1. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  2. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  3. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  4. 2024DiarizationLM: Speaker Diarization Post-Processing with Large Language Modelscited by 16
  5. 2022SepTr: Separable Transformer for Audio Spectrogram Processingcited by 15
  6. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  7. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  8. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  9. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  10. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audiocited by 10
  11. 2021Multi-turn RNN-T for streaming recognition of multi-party speechcited by 10
  12. 2023Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audiocited by 10
23 more
  1. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  2. 2021Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speechcited by 5
  3. 2022A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetingscited by 5
  4. 2024Alignment-Free Training for Transducer-based Multi-Talker ASRcited by 4
  5. 2024MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Modelscited by 4
  6. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  7. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 4
  8. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  9. 2023End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysiscited by 3
  10. 2024Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applicationscited by 3
  11. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 3
  12. 2025Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognitioncited by 3
  13. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognitioncited by 2
  14. 2023Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Networkcited by 1
  15. 2024Accuracy enhancement method for speech emotion recognition from spectrogram using temporal frequency correlation and positional information learning through knowledge transfercited by 1
  16. 2024Unsupervised Speaker Diarization in Distributed IoT Networks Using Federated Learningcited by 1
  17. 2024Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokenscited by 1
  18. 2025GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASRcited by 1
  19. 2026Xiaomi-CocktailASR-1 Technical Reportcited by 1
  20. 2024Joint Beamforming and Speaker-Attributed ASR for Real Distant-Microphone Meeting Transcriptioncited by 0
  21. 2025Whisper Speaker Identification: Leveraging Pre-Trained Multilingual Transformers for Robust Speaker Embeddingscited by 0
  22. 2026CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASRcited by 0
  23. 2026Leading-Silence Augmentation and Multi-Stage Synthetic Supervision for the Second MLC-SLM Challengecited by 0