mlsanityData to 14 Oct 2026

Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference

arXiv 2609.13205 ↗First posted 16 Aug 2026cs.IR cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Sparse long-context inference requires efficient token retrieval in both prefill and decode. Existing methods often use different retrieval strategies for the two stages, preventing one retrieval representation from being reused throughout inference. We propose Self-Indexing Attention, a training-free framework built on a shared transform-domain sign-magnitude representation. The key signs provide a reusable token-level index for grouped prefill selection and decode retrieval, while the same representation remains compatible with external KV-cache compression without separate indexer metadata. This 1-bit index enables efficient retrieval through bitwise operations widely supported by modern accelerators. At 5% attention density, Self-Indexing Attention remains close to dense attention on LongBench and RULER and achieves up to 6.1x prefill and 10.3x decode attention-operator speedups. Experiments with TurboQuant and DeepSeekV4-Flash further demonstrate compatibility with low-bit KV-cache compression and pretrained sparse-attention indexers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 49 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

40 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys15 Mar
  2. 2026Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference7 Feb
  3. 2025SALS: Sparse Attention in Latent Space for KV cache Compression28 Oct
  4. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 32

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2020Reformer: The Efficient Transformercited by 1,668
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
20 more
  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  4. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  5. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  6. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  7. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  8. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  9. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  10. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  11. 2024RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Searchcited by 48
  12. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  13. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  14. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 24
  15. 2026LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmarkcited by 19
  16. 2020Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUscited by 14
  17. 2025Inference-time sparse attention with asymmetric indexingcited by 11
  18. 2025AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularitycited by 4
  19. 2025Adamas: Hadamard Sparse Attention for Efficient Long-Context Inferencecited by 4
  20. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  2. 2026FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
  3. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  4. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  5. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  6. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
  7. 2026Simplified Sparse Attention via Gist Tokens
  8. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs