# Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-EndSpeech Recognition

arXiv: https://arxiv.org/abs/2106.09236 · HTML: https://mlsanity.com/paper/2106.09236/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2021.

## Built on by (most cited first, 9 of 9)

- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [Conversational Speech Recognition By Learning Conversation-level Characteristics](https://mlsanity.com/paper/2202.07855/) (2022)
- [Are E2E ASR models ready for an industrial usage?](https://mlsanity.com/paper/2112.12572/) (2021)
- [Similarity and Content-based Phonetic Self Attention for Speech Recognition](https://mlsanity.com/paper/2203.10252/) (2022)
- [Generalizing RNN-Transducer to Out-Domain Audio via Sparse Self-Attention Layers](https://mlsanity.com/paper/2108.10752/) (2021)
- [Deep Sparse Conformer for Speech Recognition](https://mlsanity.com/paper/2209.00260/) (2022)

## Abstract (arXiv)

End-to-end models are favored in automatic speech recognition (ASR) because of their simplified system structure and superior performance. Among these models, Transformer and Conformer have achieved state-of-the-art recognition accuracy in which self-attention plays a vital role in capturing important global information. However, the time and memory complexity of self-attention increases squarely with the length of the sentence. In this paper, a prob-sparse self-attention mechanism is introduced into Conformer to sparse the computing process of self-attention in order to accelerate inference speed and reduce space consumption. Specifically, we adopt a Kullback-Leibler divergence based sparsity measurement for each query to decide whether we compute the attention function on this query. By using the prob-sparse attention mechanism, we achieve impressively 8% to 45% inference speed-up and 15% to 45% memory usage reduction of the self-attention module of Conformer Transducer while maintaining the same level of error rate.