# Multi-Channel Transformer Transducer for Speech Recognition

arXiv: https://arxiv.org/abs/2108.12953 · HTML: https://mlsanity.com/paper/2108.12953/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [Self-Attention Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.13037/) (2019)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition](https://mlsanity.com/paper/2008.05750/) (2020)
- [Multichannel End-to-end Speech Recognition](https://mlsanity.com/paper/1703.04783/) (2017)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition](https://mlsanity.com/paper/1711.08016/) (2017)
- [Unsupervised training of neural mask-based beamforming](https://mlsanity.com/paper/1904.01578/) (2019)
- [Multi-Geometry Spatial Acoustic Modeling for Distant Speech Recognition](https://mlsanity.com/paper/1903.06539/) (2019)

## Built on by (most cited first, 8 of 8)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction](https://mlsanity.com/paper/2110.08345/) (2021)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)
- [Multi-Channel End-to-End Neural Diarization with Distributed Microphones](https://mlsanity.com/paper/2110.04694/) (2021)
- [Practice of the conformer enhanced AUDIO-VISUAL HUBERT on Mandarin and English](https://mlsanity.com/paper/2303.12187/) (2023)
- [Multi-channel multi-speaker transformer for speech recognition](https://mlsanity.com/paper/2601.02688/) (2026)
- [SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays](https://mlsanity.com/paper/2601.18037/) (2026)
- [Transformer-based Neural Beamforming for Real-Time Speech Enhancement on Smart Low-Power Hearable Devices](https://mlsanity.com/paper/2609.33755/) (2026)

## Abstract (arXiv)

Multi-channel inputs offer several advantages over single-channel, to improve the robustness of on-device speech recognition systems. Recent work on multi-channel transformer, has proposed a way to incorporate such inputs into end-to-end ASR for improved accuracy. However, this approach is characterized by a high computational complexity, which prevents it from being deployed in on-device systems. In this paper, we present a novel speech recognition model, Multi-Channel Transformer Transducer (MCTT), which features end-to-end multi-channel training, low computation cost, and low latency so that it is suitable for streaming decoding in on-device speech recognition. In a far-field in-house dataset, our MCTT outperforms stagewise multi-channel models with transformer-transducer up to 6.01% relative WER improvement (WERR). In addition, MCTT outperforms the multi-channel transformer up to 11.62% WERR, and is 15.8 times faster in terms of inference speed. We further show that we can improve the computational cost of MCTT by constraining the future and previous context in attention computations.