mlsanityData to 7 Oct 2026

Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers

arXiv 2406.16747 ↗First posted 24 Jun 2024cs.CL cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

Accommodating long sequences efficiently in autoregressive Transformers, especially within an extended context window, poses significant challenges due to the quadratic computational complexity and substantial KV memory requirements inherent in self-attention mechanisms. In this work, we introduce SPARSEK Attention, a novel sparse attention mechanism designed to overcome these computational and memory obstacles while maintaining performance. Our approach integrates a scoring network and a differentiable top-k mask operator, SPARSEK, to select a constant number of KV pairs for each query, thereby enabling gradient-based optimization. As a result, SPARSEK Attention offers linear time complexity and constant memory footprint during generation. Experimental results reveal that SPARSEK Attention outperforms previous sparse attention methods and provides significant speed improvements during both training and inference, particularly in language modeling and downstream tasks. Furthermore, our method can be seamlessly integrated into pre-trained Large Language Models (LLMs) with minimal fine-tuning, offering a practical solution for effectively managing long-range dependencies in diverse applications.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 66 of its 79 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

66 of 79 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  13. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  14. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  15. 2018Image Transformercited by 770
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  18. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  19. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  20. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  21. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  22. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  23. 2019Axial Attention in Multidimensional Transformerscited by 350
  24. 2021Random Feature Attentioncited by 350
  25. 2023Adapting Language Models to Compress Contextscited by 350
  26. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  27. 2020Sparse Sinkhorn Attentioncited by 250
  28. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  29. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  30. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  31. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  32. 2022Transformer Quality in Linear Timecited by 189
  33. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  34. 2019Sparse Sequence-to-Sequence Modelscited by 167
  35. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  36. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  37. 2019Adaptive Attention Span in Transformerscited by 148
  38. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 31; 2026: 13.

51 papers.

Sort:

By area:

  1. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  2. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  3. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  4. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  5. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  6. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  7. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  8. 2025CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attentioncited by 13
  9. 2025ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Planscited by 10
  10. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  11. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  12. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 7
38 more
  1. 2025LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactionscited by 7
  2. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  3. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  4. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  5. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsificationcited by 5
  6. 2025Cost-Optimal Grouped-Query Attention for Long-Context Modelingcited by 4
  7. 2025APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUscited by 3
  8. 2025ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphscited by 3
  9. 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 3
  10. 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
  11. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Modelscited by 3
  12. 2025Top-Theta Attention: Sparsifying Transformers by Compensated Thresholdingcited by 2
  13. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  14. 2026RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable Statecited by 2
  15. 2026Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beatcited by 2
  16. 2024Reorganizing attention-space geometry with expressive attentioncited by 1
  17. 2024You Only Use Reactive Attention Slice For Long Context Retrievalcited by 1
  18. 2024Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understandingcited by 1
  19. 2025Neural Attention Searchcited by 1
  20. 2025Crisp Attention: Regularizing Transformers via Structured Sparsitycited by 1
  21. 2025Block Sparse Flash Attentioncited by 1
  22. 2026Global Context Compression with Interleaved Vision-Text Transformationcited by 1
  23. 2026Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervisioncited by 1
  24. 2025Attention Condensation via Sparsity Induced Regularized Trainingcited by 0
  25. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0
  26. 2025SoftStep: Learning Sparse Similarity Powers Deep Neighbor-Based Regressioncited by 0
  27. 2025Lag-Relative Sparse Attention In Long Context Trainingcited by 0
  28. 2025SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformerscited by 0
  29. 2025ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoningcited by 0
  30. 2025MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matchingcited by 0
  31. 2026Towards Infinite Length Extrapolation: A Unified Approachcited by 0
  32. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 0
  33. 2026LLM-CoOpt: A Co-Design and Optimization Framework for Efficient LLM Inference on Heterogeneous Platformscited by 0
  34. 2026Chimera: Neuro-Symbolic Attention Primitives for Trustworthy Dataplane Intelligencecited by 0
  35. 2026TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchscited by 0
  36. 2026AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarizationcited by 0
  37. 2026Efficient Matrix Implementation for Rotary Position Embeddingcited by 0
  38. 2026LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoningcited by 0

50 most cited of 51 All 51 (JSON)