mlsanityData to 7 Oct 2026

MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding

arXiv 2610.04313 ↗First posted 3 Oct 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context decoding is limited by memory bandwidth, because every output token reads the KV cache of every layer. Sparse decoding reduces this cost by reading only part of the KV cache. We observe that the number of pages a query needs varies widely across KV heads, layers and steps. Fixed budgets are simple, but they are sized for demanding cases and tuned per workload; adaptive budgets follow this variation more flexibly, but existing designs pay for it with extra selection cost or training. At the kernel level, FlashAttention-3 (FA3) and FlashInfer are designed for rows of similar length: with page lists whose length differs per KV head, they either pad the lists (forfeiting much of the sparse saving), leave thread blocks unbalanced, or rely on a host-side plan that runs outside the CUDA graph. We propose MOIRA, a training-free sparse decode path in vLLM whose budget adapts per KV head and per layer. For every request, layer, KV head and step, a coverage rule keeps the smallest set of pages whose estimated attention mass reaches a fraction γ. A new kernel, self-planning attention, lets each thread block derive its own share of the work from the list lengths, so the whole decode step stays inside the CUDA graph. On an H200, at RULER's 128k context, MOIRA with γ=0.99 matches dense accuracy while reading about 30% of the pages and reduces the time per output token (TPOT) by 2.2-2.5× relative to dense FA3; with γ=0.98 it reduces TPOT by 2.7× and stays within the noise of dense. Under high serving load it raises throughput by up to 51%. These results suggest that a budget adapted per head and layer, paired with a kernel that keeps such budgets inside the CUDA graph, makes sparse decoding both flexible and fast.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 48 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

44 of 48 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference22 Sep
  2. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding16 Sep
  3. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression27 Jun
  4. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers22 Jun
  5. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference8 May
  6. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs5 Feb
  7. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference5 May
  8. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving1 Mar
  9. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  10. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs17 Feb
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  12. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
8 more
  1. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  3. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head30 Sep
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  10. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
12 more
  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  3. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  5. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  6. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  7. 2026MiniMax Sparse Attentioncited by 34
  8. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  9. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  10. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  11. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  12. 2026Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decodingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  2. 2026MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation
  3. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrieval
  4. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  5. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  6. 2026EntmaxKV: Support-Aware Decoding for Entmax Attention
  7. 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
  8. 2025Rectified Sparse Attention