# Efficient conformer-based speech recognition with linear attention

arXiv: https://arxiv.org/abs/2104.06865 · HTML: https://mlsanity.com/paper/2104.06865/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2021.

## Built on by (most cited first, 13 of 13)

- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)
- [Decoupling recognition and transcription in Mandarin ASR](https://mlsanity.com/paper/2108.01129/) (2021)
- [Effective and Efficient Mixed Precision Quantization of Speech Foundation Models](https://mlsanity.com/paper/2501.03643/) (2025)
- [One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model](https://mlsanity.com/paper/2406.10160/) (2024)
- [Are E2E ASR models ready for an industrial usage?](https://mlsanity.com/paper/2112.12572/) (2021)
- [Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations](https://mlsanity.com/paper/2503.22687/) (2025)
- [Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates](https://mlsanity.com/paper/2505.22608/) (2025)
- [Deep Sparse Conformer for Speech Recognition](https://mlsanity.com/paper/2209.00260/) (2022)
- [Attention Enhanced Citrinet for Speech Recognition](https://mlsanity.com/paper/2209.00261/) (2022)
- [Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering](https://mlsanity.com/paper/2606.11836/) (2026)

## Abstract (arXiv)

Recently, conformer-based end-to-end automatic speech recognition, which outperforms recurrent neural network based ones, has received much attention. Although the parallel computing of conformer is more efficient than recurrent neural networks, the computational complexity of its dot-product self-attention is quadratic with respect to the length of the input feature. To reduce the computational complexity of the self-attention layer, we propose multi-head linear self-attention for the self-attention layer, which reduces its computational complexity to linear order. In addition, we propose to factorize the feed forward module of the conformer by low-rank matrix factorization, which successfully reduces the number of the parameters by approximate 50% with little performance loss. The proposed model, named linear attention based conformer (LAC), can be trained and inferenced jointly with the connectionist temporal classification objective, which further improves the performance of LAC. To evaluate the effectiveness of LAC, we conduct experiments on the AISHELL-1 and LibriSpeech corpora. Results show that the proposed LAC achieves better performance than 7 recently proposed speech recognition models, and is competitive with the state-of-the-art conformer. Meanwhile, the proposed LAC has a number of parameters of only 50% over the conformer with faster training speed than the latter.