mlsanityData to 14 Oct 2026

Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies

arXiv 2511.00819 ↗First posted 2 Nov 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

In this work, we conduct a systematic analysis of Native Sparse Attention (NSA) and propose targeted improvements that enhance long-context modeling. A key insight is that alternating between local (sliding-window) and global (compression, selective) attention across layers, rather than using fixed patterns, enables more effective propagation of long-range dependencies and substantially boosts performance on long-sequence tasks. Meanwhile, we further refine NSA's branches with Latent Attention that the sliding-window branch is enhanced with Multi-head Latent Attention (MLA) while compression and selective branches adopt Group-head Latent Attention (GLA). These changes reduce KV-cache memory by 50% versus NSA while improving the model's common-sense reasoning and long-text understanding capabilities. Experiments on models from 340M to 1.3B parameters (trained on 15B and 100B tokens) show our method matches or exceeds full attention and native sparse attention in both common-sense reasoning and long-context understanding tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

35 of 49 references.

References · 35

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
23 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  5. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  10. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  12. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  13. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  14. 2021Efficient Attentions for Long Document Summarizationcited by 345
  15. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  16. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  17. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  18. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  19. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  20. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  21. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 24
  22. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 19
  23. 2024How Sparse Attention Approximates Exact Attention? Your Attention is Naturally nC-Sparsecited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  2. 2025GTA: Grouped-head latenT Attention
  3. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  4. 2025VideoNSA: Native Sparse Attention Scales Video Understanding
  5. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling
  6. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  7. 2025Lag-Relative Sparse Attention In Long Context Training
  8. 2026Sliding-window beats linear attention