mlsanityData to 14 Oct 2026

SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs

arXiv 2410.13276 ↗First posted 17 Oct 2024cs.CLcited by 124

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Attention is the cornerstone of modern Large Language Models (LLMs). Yet its quadratic complexity hinders efficiency and scalability, especially for long-context processing. A promising approach is to leverage sparsity in attention. However, existing sparsity-based solutions predominantly rely on predefined patterns or heuristics at the attention head level, struggling to adapt dynamically to different contexts efficiently. We propose SeerAttention, a simple yet effective attention mechanism that directly learns the block-level attention sparsity from the LLM itself. Inspired by the gating mechanism in Mixture of Experts (MoE), SeerAttention augments the conventional attention with a learnable gate that selectively activates important blocks within the attention map. Specifically, the gate first pools the query (Q) and key (K) tensors along the sequence dimension and processes them through learnable linear layers. The resulting matrices are then multiplied together to produce the gating scores, which are used to predict block-level attention sparsity. Combined with our block-sparse FlashAttention kernel, SeerAttention can achieve significant speedup on GPUs. When applied to pre-trained LLMs, SeerAttention only requires training the gate parameters in a lightweight self-distillation manner, allowing rapid convergence. Our evaluation results demonstrate that SeerAttention achieves better model accuracy and lower latency for long-context pre-filling compared to prior methods. Code is available at: https://github.com/microsoft/SeerAttention

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 24 of its 29 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

24 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Mistral 7Bcited by 3,164
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
11 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  7. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  8. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  9. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  10. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  11. 2021Transformer Acceleration with Dynamic Sparse Attentioncited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 61; 2026: 60.

124 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  3. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  4. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  5. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  6. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 106
  7. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  8. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  9. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  10. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  11. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  12. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
38 more
  1. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  2. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 34
  3. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  4. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  5. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  6. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 23
  7. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 22
  8. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  9. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  10. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 19
  11. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 18
  12. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  13. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  14. 2025Scaling Inference-Efficient Language Modelscited by 16
  15. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  16. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 15
  17. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  18. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  19. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  20. 2026Hierarchical Sparse Attention Done Right: Toward Infinite Context Modelingcited by 12
  21. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  22. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  23. 2025Rectified Sparse Attentioncited by 10
  24. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  25. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 10
  26. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 10
  27. 2026Speed3R: Sparse Feed-forward 3D Reconstruction Modelscited by 10
  28. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 9
  29. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 9
  30. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  31. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compressioncited by 7
  32. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Servingcited by 7
  33. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsificationcited by 7
  34. 2025Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inferencecited by 7
  35. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inferencecited by 7
  36. 2025RWKV-X: A Linear Complexity Hybrid Language Modelcited by 6
  37. 2025Trainable Dynamic Mask Sparse Attentioncited by 6
  38. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 6

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
  2. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  3. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
  4. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
  5. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  6. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
  7. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
  8. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference