mlsanityData to 7 Oct 2026

KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

arXiv 2605.09735 ↗First posted 10 May 2026cs.AR cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Static-graph LLM decoders provide predictable launches, fixed tensor shapes, and low submission overhead, but online decoding exposes highly irregular KV-cache behavior: request lengths differ, EOS events arrive asynchronously, and logical histories fragment over time. Dynamic runtimes recover flexibility through paged KV management and step-level scheduling, while static-graph executors often over-reserve memory and suffer burst-time latency outliers. This paper studies whether much of this variability can be absorbed below a fixed decode interface. We present KV-RM, a runtime design that regularizes KV-cache movement beneath a static-graph LLM decoder. KV-RM decouples logical KV histories from physical storage, tracks active KV state through a block pager, and materializes each decode step through a single committed descriptor. A merge-staged transport path coalesces non-contiguous KV mappings into a small number of large transfer groups before a fixed-shape attention kernel consumes them. Optional bounded far-history summaries can be enabled under the same interface, but the core design does not depend on them. On a 2-GPU NVIDIA A100 node, KV-RM improves mixed-length decoding throughput and tail latency relative to a static-graph baseline, reduces reserved KV memory across workload families, and removes severe burst-time latency spikes under production-trace replay. These results suggest that KV-cache movement, rather than kernel shape, can be an effective boundary for recovering runtime flexibility in static-graph LLM serving.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 45 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

25 of 45 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 17

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
5 more
  1. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  3. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  4. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  5. 2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  2. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
  3. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  4. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  5. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  6. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch
  7. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  8. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving