Exploring Attention Map Reuse for Efficient Transformer Neural Networks
Not on the map. Why?
Abstract (arXiv)
Transformer-based deep neural networks have achieved great success in various sequence applications due to their powerful ability to model long-range dependency. The key module of Transformer is self-attention (SA) which extracts features from the entire sequence regardless of the distance between positions. Although SA helps Transformer performs particularly well on long-range tasks, SA requires quadratic computation and memory complexity with the input sequence length. Recently, attention map reuse, which groups multiple SA layers to share one attention map, has been proposed and achieved significant speedup for speech recognition models. In this paper, we provide a comprehensive study on attention map reuse focusing on its ability to accelerate inference. We compare the method with other SA compression techniques and conduct a breakdown analysis of its advantages for a long sequence. We demonstrate the effectiveness of attention map reuse by measuring the latency on both CPU and GPU platforms.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 29 Jan 2023.
Builds on
28 of 35 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2016Layer Normalizationcited by 5,579
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2020Reformer: The Efficient Transformercited by 1,658
- 2020Rethinking Attention with Performerscited by 1,584
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2020Efficient Transformers: A Surveycited by 784
16 more
- 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
- 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
- 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
- 2020Fast Transformers with Clustered Attentioncited by 123
- 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
- 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
- 2019Sharing Attention Weights for Fast Transformercited by 34
- 2021Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimizationcited by 27
- 2020Simplified Self-Attention for Transformer-based End-to-End Speech Recognitioncited by 20
- 2021Pushing the Limits of Non-Autoregressive Speech Recognitioncited by 18
- 2020Know What You Don't Need: Single-Shot Meta-Pruning for Attention Headscited by 16
- 2021LazyFormer: Self Attention with Lazy Updatecited by 16
- 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15
- 2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
- 2021Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T modelscited by 3
Built on by
Citing papers per year: 2024: 3.
3 papers.