mlsanityData to 7 Oct 2026

Leveraging redundancy in attention with Reuse Transformers

arXiv 2110.06821 ↗First posted 13 Oct 2021cs.LG cs.CL cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision. However, a typical Transformer model computes such pairwise attention scores repeatedly for the same sequence, in multiple heads in multiple layers. We systematically analyze the empirical similarity of these scores across heads and layers and find them to be considerably redundant, especially adjacent layers showing high similarity. Motivated by these findings, we propose a novel architecture that reuses attention scores computed in one layer in multiple subsequent layers. Experiments on a number of standard benchmarks show that reusing attention delivers performance equivalent to or better than standard transformers, while reducing both compute and memory usage.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

30 of 33 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
18 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  4. 2019Attention is not Explanationcited by 916
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2020Efficient Transformers: A Surveycited by 784
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2019Attention is not not Explanationcited by 629
  11. 2018Universal Transformerscited by 613
  12. 2019Is Attention Interpretable?cited by 387
  13. 2021Random Feature Attentioncited by 350
  14. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  15. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  16. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 42
  17. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  18. 2021Eigen Analysis of Self-Attention and its Reconstruction from Partial Computationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 9; 2025: 12; 2026: 4.

33 papers.

Sort:

By area:

  1. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  2. 2022On The Computational Complexity of Self-Attentioncited by 137
  3. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  4. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 35
  5. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  6. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  7. 2022Visualizing and Understanding Patch Interactions in Vision Transformercited by 26
  8. 2024MLP Can Be A Good Transformer Learnercited by 16
  9. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  10. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similaritycited by 11
  11. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  12. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 9
21 more
  1. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  2. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  3. 2025Emergent Extreme-View Geometry in 3D Foundation Modelscited by 5
  4. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  5. 2024Understanding Neural Network Binarization with Forward and Backward Proximal Quantizerscited by 4
  6. 2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3
  7. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systemscited by 2
  8. 2024EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Modelscited by 2
  9. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helpscited by 2
  10. 2025Beyond Parameters: Exploring Virtual Logic Depth for Scaling Lawscited by 2
  11. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  12. 2023Fast Sampling Through The Reuse Of Attention Maps In Diffusion Modelscited by 1
  13. 2025MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 1
  14. 2025Scalable Spatiotemporal Inference with Biased Scan Attention Transformer Neural Processescited by 1
  15. 2025Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generationcited by 1
  16. 2026Low-Rank Key Value Attentioncited by 1
  17. 2023Extracting Interpretable Local and Global Representations from Attention on Time Seriescited by 0
  18. 2025Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inferencecited by 0
  19. 2025FAR: Function-preserving Attention Replacement for IMC-friendly Inferencecited by 0
  20. 2026From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillationcited by 0
  21. 2026Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformerscited by 0