# MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition

arXiv: https://arxiv.org/abs/1910.06522 · HTML: https://mlsanity.com/paper/1910.06522/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2019.

## Built on by (most cited first, 30 of 44)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition](https://mlsanity.com/paper/2110.04934/) (2021)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [Adapting Multi-Lingual ASR Models for Handling Multiple Talkers](https://mlsanity.com/paper/2305.18747/) (2023)
- [A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio](https://mlsanity.com/paper/2107.02852/) (2021)
- [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Advancing Multi-talker ASR Performance with Large Language Models](https://mlsanity.com/paper/2408.17431/) (2024)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding](https://mlsanity.com/paper/2207.09514/) (2022)
- [Perceive and predict: self-supervised speech representation based loss functions for speech enhancement](https://mlsanity.com/paper/2301.04388/) (2023)
- [Multi-Channel Multi-Speaker ASR Using 3D Spatial Feature](https://mlsanity.com/paper/2111.11023/) (2021)
- [VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition](https://mlsanity.com/paper/2209.04974/) (2022)
- [Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition](https://mlsanity.com/paper/2409.00815/) (2024)
- [Continuous Speech Separation with Ad Hoc Microphone Arrays](https://mlsanity.com/paper/2103.02378/) (2021)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR](https://mlsanity.com/paper/2305.13716/) (2023)
- [Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder](https://mlsanity.com/paper/2508.20474/) (2025)
- [Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation](https://mlsanity.com/paper/2204.03232/) (2022)
- [Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition](https://mlsanity.com/paper/2307.02909/) (2023)
- [LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization](https://mlsanity.com/paper/2409.00819/) (2024)
- [Audio-visual multi-channel speech separation, dereverberation and recognition](https://mlsanity.com/paper/2204.01977/) (2022)
- [The Effect of Spoken Language on Speech Enhancement using Self-Supervised Speech Representation Loss Functions](https://mlsanity.com/paper/2307.14502/) (2023)
- [End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis](https://mlsanity.com/paper/2310.10106/) (2023)
- [SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition](https://mlsanity.com/paper/2401.18045/) (2024)
- [Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition](https://mlsanity.com/paper/2506.14973/) (2025)
- [A Time-domain Real-valued Generalized Wiener Filter for Multi-channel Neural Separation Systems](https://mlsanity.com/paper/2112.03533/) (2021)

## Abstract (arXiv)

Recently, the end-to-end approach has proven its efficacy in monaural multi-speaker speech recognition. However, high word error rates (WERs) still prevent these systems from being used in practical applications. On the other hand, the spatial information in multi-channel signals has proven helpful in far-field speech recognition tasks. In this work, we propose a novel neural sequence-to-sequence (seq2seq) architecture, MIMO-Speech, which extends the original seq2seq to deal with multi-channel input and multi-channel output so that it can fully model multi-channel multi-speaker speech separation and recognition. MIMO-Speech is a fully neural end-to-end framework, which is optimized only via an ASR criterion. It is comprised of: 1) a monaural masking network, 2) a multi-source neural beamformer, and 3) a multi-output speech recognition model. With this processing, the input overlapped speech is directly mapped to text sequences. We further adopted a curriculum learning strategy, making the best use of the training set to improve the performance. The experiments on the spatialized wsj1-2mix corpus show that our model can achieve more than 60% WER reduction compared to the single-channel system with high quality enhanced signals (SI-SDR = 23.1 dB) obtained by the above separation function.