# State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions

arXiv: https://arxiv.org/abs/1910.00716 · HTML: https://mlsanity.com/paper/1910.00716/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 11 of 11)

- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends](https://mlsanity.com/paper/2402.15490/) (2024)
- [MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription](https://mlsanity.com/paper/2108.02625/) (2021)
- [Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR](https://mlsanity.com/paper/2405.14259/) (2024)
- [The RoyalFlush System of Speech Recognition for M2MeT Challenge](https://mlsanity.com/paper/2202.01614/) (2022)
- [Deformable TDNN with adaptive receptive fields for speech recognition](https://mlsanity.com/paper/2104.14791/) (2021)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [Predicting Opioid Use Disorder from Longitudinal Healthcare Data using Multi-stream Transformer](https://mlsanity.com/paper/2103.08800/) (2021)
- [EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Accelerators](https://mlsanity.com/paper/2202.02310/) (2022)
- [Coarse-To-Fine And Cross-Lingual ASR Transfer](https://mlsanity.com/paper/2109.00916/) (2021)

## Abstract (arXiv)

Self-attention has been a huge success for many downstream tasks in NLP, which led to exploration of applying self-attention to speech problems as well. The efficacy of self-attention in speech applications, however, seems not fully blown yet since it is challenging to handle highly correlated speech frames in the context of self-attention. In this paper we propose a new neural network model architecture, namely multi-stream self-attention, to address the issue thus make the self-attention mechanism more effective for speech recognition. The proposed model architecture consists of parallel streams of self-attention encoders, and each stream has layers of 1D convolutions with dilated kernels whose dilation rates are unique given stream, followed by a self-attention layer. The self-attention mechanism in each stream pays attention to only one resolution of input speech frames and the attentive computation can be more efficient. In a later stage, outputs from all the streams are concatenated then linearly projected to the final embedding. By stacking the proposed multi-stream self-attention encoder blocks and rescoring the resultant lattices with neural network language models, we achieve the word error rate of 2.2% on the test-clean dataset of the LibriSpeech corpus, the best number reported thus far on the dataset.