mlsanityData to 14 Oct 2026

DoSA: Dots Sparse Attention

arXiv 2610.13990 ↗First posted 26 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Sparse attention visits few keys, yet finding them can still require scoring every key with every indexer head. We introduce DoSA (Dots Sparse Attention), a lossless method for making this computation conditional while preserving exact token selection. Its central observation is that Top-K selection requires sufficient evidence to reject a candidate, not necessarily its full score. A seed set supplies a lower bound on the winning threshold. Nonnegative head projections and joint residual bounds upper-bound positive contributions, while activation-informed aggregation bounds the magnitude subtracted by negative heads. The certificate directs full scoring toward candidates that remain eligible; exact-score filtering further reduces writeback and selection work. We establish the certificate and its exact-selection conditions, and analyze how geometry and selection margins govern its effectiveness. Our GPU implementation accelerates the complete Indexer by up to 2.74 times over SGLang and 1.69 times over LiteTopK across English, Chinese, and code workloads. End-to-end evaluation on GLM-5.2 and DeepSeek-V3.2 achieves up to 1.94 times and 1.50 times time-to-first-token speedups over the two baselines, respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

19 of 26 references.

References · 19

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  4. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  7. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  8. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  9. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compressioncited by 76
  10. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  11. 2026MiniMax Sparse Attentioncited by 44
  12. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
7 more
  1. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  2. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 12
  3. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inferencecited by 7
  4. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inferencecited by 5
  5. 2026Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell via Temporal Correlationcited by 3
  6. 2026PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attentioncited by 3
  7. 2026LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attentioncited by 2

Built on by

Not cited yet.