mlsanityData to 7 Oct 2026

PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding

arXiv 2609.34555 ↗First posted 28 Sep 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization also fits: Serving systems and request scheduling

Abstract (arXiv)

Long-context LLM serving is increasingly bottlenecked by decode, where large KV caches limit batch size and underutilize GPUs. Sparse KV cache offloading expands effective capacity by storing most historical KV blocks in CPU DRAM and recalling only selected blocks on demand. However, we find that existing offloading systems shift the bottleneck to CPU-GPU recall I/O: recall volume varies widely across layers, decode steps and requests, while headwise sparse selection fragments recalls into many small PCIe transfers. This paper presents PulseInfer, an I/O-centric sparse KV cache offloading system. PulseInfer hides variable recall latency with interruptible layer-wise scheduling, adapts offloading decisions with IO-Adaptive Offloading Admission, and coalesces fragmented transfers using SoloHead sparse selection and a gather-scatter I/O engine. Implemented on SGLang, PulseInfer improves decode throughput by up to 4.7x over SGLang and 2.6x over the best existing offloading baseline, while reducing TPOT by up to 76% and preserving near-lossless accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 48 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

30 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  2. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  3. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation26 Nov
  4. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
11 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  3. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  5. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  7. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  9. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  10. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  11. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  2. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  3. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  4. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  5. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  6. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  7. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
  8. 2025NOSA: Native and Offloadable Sparse Attention