mlsanityData to 7 Oct 2026

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

arXiv 2502.11089 ↗First posted 16 Feb 2025cs.CL cs.AI cs.LGcited by 391

Not on the map. Why?

Cited by 102 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling. NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. Our approach advances sparse attention design with two key innovations: (1) We achieve substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) We enable end-to-end training, reducing pretraining computation without sacrificing model performance. As shown in Figure 1, experiments show the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves substantial speedups over Full Attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle.

Key facts. Cited by 391 papers in the map (data through 7 Oct 2026). Builds on 42 of its 48 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

42 of 48 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
30 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  13. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  14. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  15. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  16. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  17. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  18. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  19. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  20. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  21. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  22. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  23. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  24. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  25. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  26. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  27. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  28. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
  29. 2024LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Modelscited by 17
  30. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 168; 2026: 223.

391 papers.

Sort:

By area:

  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  3. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  4. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  7. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  9. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  10. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  11. 2025Efficient Reasoning Models: A Surveycited by 93
  12. 2025Mixture of Contexts for Long Video Generationcited by 90
38 more
  1. 2025LongCat-Video Technical Reportcited by 73
  2. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  3. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  4. 2025End-to-End Test-Time Training for Long Contextcited by 53
  5. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  6. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  7. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  8. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  9. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  10. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  11. 2025VMoBA: Mixture-of-Block Attention for Video Diffusion Modelscited by 35
  12. 2026MiniMax Sparse Attentioncited by 34
  13. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 33
  14. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  15. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  16. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  17. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  18. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  19. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  20. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  21. 2025Inference-Time Hyper-Scaling with KV Cache Compressioncited by 28
  22. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  23. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  24. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  25. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  26. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 22
  27. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  28. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  29. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  30. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  31. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  32. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  33. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  34. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  35. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  36. 2025Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Lightcited by 16
  37. 2025MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationcited by 16
  38. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 16

50 most cited of 391 All 391 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  2. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
  3. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  4. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
  5. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
  6. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  7. 2026STS: Efficient Sparse Attention with Speculative Token Sparsity
  8. 2024Loki: Low-rank Keys for Efficient Sparse Attention