mlsanityData to 7 Oct 2026

Selective Attention Improves Transformer

arXiv 2410.02703 ↗First posted 3 Oct 2024cs.CL cs.AI cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Unneeded elements in the attention's context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces attention to unneeded elements. Selective attention consistently improves language modeling and downstream task performance in a variety of model sizes and context lengths. For example, transformers trained with the language modeling objective on C4 with selective attention perform language modeling equivalently to standard transformers with ~2X more heads and parameters in their attention modules. Selective attention also allows decreasing the size of the attention's context buffer, leading to meaningful reductions in the memory and compute requirements during inference. For example, transformers trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and 47X less memory for their attention module, respectively, when equipped with selective attention, as those without selective attention, with the same validation perplexity.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 43 of its 49 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

43 of 49 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
31 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  13. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  14. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  15. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  16. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  17. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  18. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  19. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  20. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  21. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  22. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  23. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  24. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  25. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  26. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  27. 2021Sparse is Enough in Scaling Transformerscited by 86
  28. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  29. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10
  30. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokenscited by 8
  31. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 8.

26 papers.

Sort:

By area:

  1. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  2. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  3. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  4. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  5. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  6. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  7. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  8. 2025Prompt Repetition Improves Non-Reasoning LLMscited by 23
  9. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 15
  10. 2024KV Shifting Attention Enhances Language Modelingcited by 7
  11. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  12. 2025Adaptive Computation Pruning for the Forgetting Transformercited by 6
14 more
  1. 2026The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modelingcited by 3
  2. 2026The Topological Trouble With Transformerscited by 3
  3. 2026Memory for Large Language Modelscited by 2
  4. 2025Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modelingcited by 1
  5. 2025Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Modelscited by 1
  6. 2026Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsitycited by 1
  7. 2025Causal Attention with Lookahead Keyscited by 0
  8. 2025Dynamic Relational Priming Improves Transformer in Multivariate Time Seriescited by 0
  9. 2025Transformer Modeling for Both Scalability and Performance in Multivariate Time Seriescited by 0
  10. 2025Can We Go Beyond Visual Features? Neural Tissue Relation Modeling for Relational Graph Analysis in Non-Melanoma Skin Histologycited by 0
  11. 2026Attention-guided Evidence Grounding for Spoken Question Answeringcited by 0
  12. 2026StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-kcited by 0
  13. 2026Abstention and Noise Filtering: Two Missing Primitives of Softmax Attentioncited by 0
  14. 2026MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometriescited by 0