# Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention

arXiv: https://arxiv.org/abs/2010.12155 · HTML: https://mlsanity.com/paper/2010.12155/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Efficient Transformer for Direct Speech Translation](https://mlsanity.com/paper/2107.03069/) (2021)
- [Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention](https://mlsanity.com/paper/2103.15722/) (2021)
- [Conformer-based End-to-end Speech Recognition With Rotary Position Embedding](https://mlsanity.com/paper/2107.05907/) (2021)
- [Similarity and Content-based Phonetic Self Attention for Speech Recognition](https://mlsanity.com/paper/2203.10252/) (2022)
- [Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognition](https://mlsanity.com/paper/2604.27533/) (2026)

## Abstract (arXiv)

Recently, several studies reported that dot-product selfattention (SA) may not be indispensable to the state-of-theart Transformer models. Motivated by the fact that dense synthesizer attention (DSA), which dispenses with dot products and pairwise interactions, achieved competitive results in many language processing tasks, in this paper, we first propose a DSA-based speech recognition, as an alternative to SA. To reduce the computational complexity and improve the performance, we further propose local DSA (LDSA) to restrict the attention scope of DSA to a local range around the current central frame for speech recognition. Finally, we combine LDSA with SA to extract the local and global information simultaneously. Experimental results on the Ai-shell1 Mandarine speech recognition corpus show that the proposed LDSA-Transformer achieves a character error rate (CER) of 6.49%, which is slightly better than that of the SA-Transformer. Meanwhile, the LDSA-Transformer requires less computation than the SATransformer. The proposed combination method not only achieves a CER of 6.18%, which significantly outperforms the SA-Transformer, but also has roughly the same number of parameters and computational complexity as the latter. The implementation of the multi-head LDSA is available at https://github.com/mlxu995/multihead-LDSA.