mlsanityData to 7 Oct 2026

StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

arXiv 2605.02568 ↗First posted 4 May 2026cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

DeepSeek-V3.2 and V4 introduce Compressed Sparse Attention (CSA): a lightning indexer (a learned scoring projection over compressed keys) scores them, the top-k are selected per query, and a sparse attention kernel reads only those. Public CSA implementations materialize a [B, S, H_I, T] FP32 score tensor before the top-k reduction. With H_I=64 indexer heads and the V4-Flash compression ratio m=4, that intermediate is 256 GB at sequence length S=65,536, exceeding any single-GPU high-bandwidth-memory (HBM) budget. We present StreamIndex, a Triton implementation of the CSA pipeline whose central component is a chunked partition-merge top-k driver that never materializes the full intermediate. On synthetic-but-realistic V4-shaped inputs at the indexer-step (layer) level on a single NVIDIA H200, the materialize path runs out of memory (OOMs) at S=65,536 with V4-Flash dimensions; StreamIndex runs the same indexer to S=1,048,576 with 6.21 GB peak HBM, a 32x regime extension. Set-overlap recall against the materialize ground truth is bit-exact at small S where both fit; across three 5-point design-space sweeps (chunk size, key-tile size, top-k), mean recall rounds to 1.0000 with min recall at least 0.9980 in every cell. The chunked driver composes with TileLang's pipelined attention kernel: at S=262,144 with V4-Flash dimensions, the materialize indexer paired with TileLang attention OOMs while the chunked indexer paired with the same attention runs in 1.97 s at 18.56 GB peak. Our contribution targets the indexer step; we make no claim of a faster attention kernel or of real-checkpoint end-to-end behavior. Code: https://github.com/RightNow-AI/StreamIndex.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

30 of 38 references.

References · 30

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
18 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  6. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  12. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  13. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  14. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  15. 2022FP8 Formats for Deep Learningcited by 187
  16. 2023Microscaling Data Formats for Deep Learningcited by 132
  17. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 42
  18. 2024Selective Attention Improves Transformercited by 26

Built on by

Not cited yet.