mlsanityData to 13 Oct 2026

MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition

arXiv 1910.06522 ↗First posted 15 Oct 2019eess.AS cs.CL cs.SDcited by 44

Not on the map. Why?

Abstract (arXiv)

Recently, the end-to-end approach has proven its efficacy in monaural multi-speaker speech recognition. However, high word error rates (WERs) still prevent these systems from being used in practical applications. On the other hand, the spatial information in multi-channel signals has proven helpful in far-field speech recognition tasks. In this work, we propose a novel neural sequence-to-sequence (seq2seq) architecture, MIMO-Speech, which extends the original seq2seq to deal with multi-channel input and multi-channel output so that it can fully model multi-channel multi-speaker speech separation and recognition. MIMO-Speech is a fully neural end-to-end framework, which is optimized only via an ASR criterion. It is comprised of: 1) a monaural masking network, 2) a multi-source neural beamformer, and 3) a multi-output speech recognition model. With this processing, the input overlapped speech is directly mapped to text sequences. We further adopted a curriculum learning strategy, making the best use of the training set to improve the performance. The experiments on the spatialized wsj1-2mix corpus show that our model can achieve more than 60% WER reduction compared to the single-channel system with high quality enhanced signals (SI-SDR = 23.1 dB) obtained by the above separation function.

Key facts. Cited by 44 papers in the map (data through 13 Oct 2026). First posted on arXiv 15 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 12; 2022: 13; 2023: 8; 2024: 6; 2025: 4; 2026: 1.

44 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2023End-to-End Speech Recognition: A Surveycited by 128
  3. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  4. 2021End-to-End Multi-Channel Transformer for Speech Recognitioncited by 17
  5. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  6. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  7. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  8. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audiocited by 10
  9. 2021VarArray: Array-Geometry-Agnostic Continuous Speech Separationcited by 10
  10. 2022Simulating realistic speech overlaps improves multi-talker ASRcited by 10
  11. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  12. 2021Multi-Channel Transformer Transducer for Speech Recognitioncited by 8
32 more
  1. 2022ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understandingcited by 8
  2. 2023Perceive and predict: self-supervised speech representation based loss functions for speech enhancementcited by 8
  3. 2021Multi-Channel Multi-Speaker ASR Using 3D Spatial Featurecited by 7
  4. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
  5. 2024Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognitioncited by 7
  6. 2021Continuous Speech Separation with Ad Hoc Microphone Arrayscited by 5
  7. 2022A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetingscited by 5
  8. 2023BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASRcited by 5
  9. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 5
  10. 2022Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separationcited by 4
  11. 2023Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognitioncited by 4
  12. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 4
  13. 2022Audio-visual multi-channel speech separation, dereverberation and recognitioncited by 3
  14. 2023The Effect of Spoken Language on Speech Enhancement using Self-Supervised Speech Representation Loss Functionscited by 3
  15. 2023End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysiscited by 3
  16. 2024SpeechComposer: Unifying Multiple Speech Tasks with Prompt Compositioncited by 3
  17. 2025Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognitioncited by 3
  18. 2021A Time-domain Real-valued Generalized Wiener Filter for Multi-channel Neural Separation Systemscited by 1
  19. 2022Exploiting Single-Channel Speech for Multi-Channel End-to-End Speech Recognition: A Comparative Studycited by 1
  20. 2022End-to-End Multi-speaker ASR with Independent Vector Analysiscited by 1
  21. 2022Customizable End-to-end Optimization of Online Neural Network-supported Dereverberation for Hearing Devicescited by 1
  22. 2024Transcription-Free Fine-Tuning of Speech Separation Models for Noisy and Reverberant Multi-Speaker Automatic Speech Recognitioncited by 1
  23. 2021Blind Speech Separation and Dereverberation using Neural Beamformingcited by 0
  24. 2021Robustness of end-to-end Automatic Speech Recognition Models -- A Case Study using Mozilla DeepSpeechcited by 0
  25. 2021ChannelAugment: Improving generalization of multi-channel ASR by training with input channel randomizationcited by 0
  26. 2022A neural network-supported two-stage algorithm for lightweight dereverberation on hearing devicescited by 0
  27. 2022Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarizationcited by 0
  28. 2023Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representationcited by 0
  29. 2024Joint Beamforming and Speaker-Attributed ASR for Real Distant-Microphone Meeting Transcriptioncited by 0
  30. 2025SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognitioncited by 0
  31. 2025Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarioscited by 0
  32. 2026Multi-channel multi-speaker transformer for speech recognitioncited by 0