mlsanityData to 7 Oct 2026

KVBuffer: IO-aware Serving for Linear Attention

arXiv 2605.19049 ↗First posted 18 May 2026cs.LG cs.AIcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length. However, existing serving systems typically serve linear attention by recurrently computing and updating a large linear attention state in every decoding step. Since the state is much larger than the per-token key and value, recurrent decoding incurs substantial memory access and becomes inefficient for serving linear attention. In this paper, we propose KVBuffer, an IO-aware serving mechanism for linear attention. By buffering recent keys and values, KVBuffer enables serving systems to compute linear attention outputs in more flexible and memory-efficient ways. For decoding, KVBuffer enables chunkwise computation, which reduces average memory access and decoding latency by deferring state updates and applying them in batch. For speculative decoding, KVBuffer verifies draft tokens in parallel and avoids storing temporary states. For short contexts, KVBuffer computes attention outputs directly from buffered keys and values, without creating or updating the linear attention state. We implement KVBuffer in SGLang for Qwen3-Next. Our evaluations show that KVBuffer can reduce linear attention decoding latency by up to 45.17% and increase the maximum number of serving requests by 5x for speculative decoding when verifying four draft tokens.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 25 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

20 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter16 Apr
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 18

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  10. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  12. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
6 more
  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  3. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  4. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  5. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  6. 2025STree: Speculative Tree Decoding for Hybrid State-Space Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decodingcited by 1
  2. 2026Low-Bit Recurrent States in Hybrid Language Modelscited by 0
  3. 2026SketchSSM: Write to the Full State, Read from a Compact Sketchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference
  2. 2026Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
  3. 2026MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation
  4. 2026SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
  5. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
  6. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  7. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  8. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference