mlsanityData to 7 Oct 2026

Predicting Attention Sparsity in Transformers

arXiv 2109.12188 ↗First posted 24 Sep 2021cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformers' quadratic complexity with respect to the input sequence length has motivated a body of work on efficient sparse approximations to softmax. An alternative path, used by entmax transformers, consists of having built-in exact sparse attention; however this approach still requires quadratic computation. In this paper, we propose Sparsefinder, a simple model trained to identify the sparsity pattern of entmax attention before computing it. We experiment with three variants of our method, based on distances, quantization, and clustering, on two tasks: machine translation (attention in the decoder) and masked language modeling (encoder-only). Our work provides a new angle to study model efficiency by doing extensive analysis of the tradeoff between the sparsity and recall of the predicted attention graph. This allows for detailed comparison between different models along their Pareto curves, important to guide future benchmarks for sparse attention models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

29 of 42 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
17 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  5. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  6. 2020Sparse Sinkhorn Attentioncited by 250
  7. 2019Sparse Sequence-to-Sequence Modelscited by 167
  8. 2019Adaptively Sparse Transformerscited by 154
  9. 2020Fast Transformers with Clustered Attentioncited by 123
  10. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  11. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  12. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  13. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  14. 2020SMYRF: Efficient Attention using Asymmetric Clusteringcited by 43
  15. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 42
  16. 2021Sparse Attention with Linear Unitscited by 41
  17. 2020Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encodingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 4; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  2. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  3. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  4. 2022ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 41
  5. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  6. 2025Long-Context Generalization with Sparse Attentioncited by 18
  7. 2025AdaSplash: Adaptive Sparse Flash Attentioncited by 12
  8. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  9. 2024Conditional computation in neural networks: principles and research trendscited by 10
  10. 2024Reducing Reasoning Costs: The Path of Optimization for Chain of Thought via Sparse Attention Mechanismcited by 4
  11. 2026Improving Sparse Autoencoder with Dynamic Attentioncited by 2
  12. 2026Perspectives on Tsallis Statistics for Artificial Intelligencecited by 1
1 more
  1. 2025BiSparse-AAS: Bilinear Sparse Attention and Adaptive Spans Framework for Scalable and Efficient Text Summarizationcited by 0