# End-to-End Multi-Channel Transformer for Speech Recognition

arXiv: https://arxiv.org/abs/2102.03951 · HTML: https://mlsanity.com/paper/2102.03951/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 8 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Channel-Attention Dense U-Net for Multichannel Speech Enhancement](https://mlsanity.com/paper/2001.11542/) (2020)
- [Multichannel End-to-end Speech Recognition](https://mlsanity.com/paper/1703.04783/) (2017)
- [Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition](https://mlsanity.com/paper/1711.08016/) (2017)
- [Frequency Domain Multi-channel Acoustic Modeling for Distant Speech Recognition](https://mlsanity.com/paper/1903.05299/) (2019)
- [Unsupervised training of neural mask-based beamforming](https://mlsanity.com/paper/1904.01578/) (2019)
- [Multi-Geometry Spatial Acoustic Modeling for Distant Speech Recognition](https://mlsanity.com/paper/1903.06539/) (2019)

## Built on by (most cited first, 17 of 17)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers](https://mlsanity.com/paper/2112.09685/) (2021)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition](https://mlsanity.com/paper/2206.15408/) (2022)
- [Self-Attention Channel Combinator Frontend for End-to-End Multichannel Far-field Speech Recognition](https://mlsanity.com/paper/2109.04783/) (2021)
- [Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognition](https://mlsanity.com/paper/2109.08744/) (2021)
- [Speech enhancement with frequency domain auto-regressive modeling](https://mlsanity.com/paper/2309.13537/) (2023)
- [End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis](https://mlsanity.com/paper/2310.10106/) (2023)
- [Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention](https://mlsanity.com/paper/2204.03479/) (2022)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [Transformer-Driven Neural Beamforming with Imperfect CSI in Urban Macro Wireless Channels](https://mlsanity.com/paper/2504.11667/) (2025)
- [Research on an improved Conformer end-to-end Speech Recognition Model with R-Drop Structure](https://mlsanity.com/paper/2306.08329/) (2023)
- [Automated Sperm Assessment Framework and Neural Network Specialized for Sperm Video Recognition](https://mlsanity.com/paper/2311.05927/) (2023)
- [RelUNet: Relative Channel Fusion U-Net for Multichannel Speech Enhancement](https://mlsanity.com/paper/2410.05019/) (2024)
- [Multi-channel multi-speaker transformer for speech recognition](https://mlsanity.com/paper/2601.02688/) (2026)
- [Multi-Level Embedding Conformer Framework for Bengali Automatic Speech Recognition](https://mlsanity.com/paper/2601.09710/) (2025)
- [SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays](https://mlsanity.com/paper/2601.18037/) (2026)

## Abstract (arXiv)

Transformers are powerful neural architectures that allow integrating different modalities using attention mechanisms. In this paper, we leverage the neural transformer architectures for multi-channel speech recognition systems, where the spectral and spatial information collected from different microphones are integrated using attention layers. Our multi-channel transformer network mainly consists of three parts: channel-wise self attention layers (CSA), cross-channel attention layers (CCA), and multi-channel encoder-decoder attention layers (EDA). The CSA and CCA layers encode the contextual relationship within and between channels and across time, respectively. The channel-attended outputs from CSA and CCA are then fed into the EDA layers to help decode the next token given the preceding ones. The experiments show that in a far-field in-house dataset, our method outperforms the baseline single-channel transformer, as well as the super-directive and neural beamformers cascaded with the transformers.