mlsanityData to 14 Oct 2026

Adaptively Sparse Transformers

arXiv 1909.00015 ↗First posted 30 Aug 2019cs.CL stat.MLcited by 155

Not on the map. Why?

Abstract (arXiv)

Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with α-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the α parameter -- which controls the shape and sparsity of α-entmax -- allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations.

Key facts. Cited by 155 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 24; 2023: 16; 2024: 32; 2025: 27; 2026: 24.

155 papers.

Sort:

By area:

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  5. 2022Transformers in Medical Imaging: A Surveycited by 245
  6. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  7. 2023Learning A Sparse Transformer Network for Effective Image Derainingcited by 205
  8. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  9. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  10. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  12. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
38 more
  1. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  2. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  3. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  4. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  5. 2024Attention Heads of Large Language Models: A Surveycited by 67
  6. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 65
  7. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  8. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  9. 2021A Survey on Green Deep Learningcited by 52
  10. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  11. 2023On Sparse Modern Hopfield Modelcited by 42
  12. 2021Sparse Attention with Linear Unitscited by 41
  13. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  14. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  15. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  16. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  17. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  18. 2022SIMPLE: A Gradient Estimator for k-Subset Samplingcited by 34
  19. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  20. 2021Neural Collaborative Graph Machines for Table Structure Recognitioncited by 30
  21. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  22. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  23. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  24. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  25. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 24
  26. 2025The Dual-Route Model of Inductioncited by 24
  27. 2021SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentationcited by 23
  28. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  29. 2025Long-Context Generalization with Sparse Attentioncited by 21
  30. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  31. 2023From Latent Graph to Latent Topology Inference: Differentiable Cell Complex Modulecited by 19
  32. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  33. 2023Transformers in Healthcare: A Surveycited by 17
  34. 2024Attention layers provably solve single-location regressioncited by 17
  35. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 16
  36. 2024Nonparametric Modern Hopfield Modelscited by 16
  37. 2024LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Modelscited by 16
  38. 2021The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERTcited by 15

50 most cited of 155 All 155 (JSON)