mlsanityData to 7 Oct 2026

Efficient Remote KV Cache Reuse with GPU-native Video Codec

arXiv 2602.09725 ↗First posted 10 Feb 2026cs.DC cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Remote KV cache reuse fetches KV cache for identical contexts from remote storage, avoiding recomputation, accelerating LLM inference. While it excels in high-speed networks, its performance degrades significantly in bandwidth-limited scenarios. Recent studies address this by transmitting KV caches in compressed form, but the associated heavyweight decompression counteracts the KV reuse benefits. In this paper, we propose an efficient and widely deployable remote KV cache reuse solution that leverages GPU-native video codecs. Our system, KVCodec, enables effective KV cache coding with two techniques. The codec-friendly tensor layout compresses the KV cache in a highly compact video format, enabling fast transmission. The efficient KV fetcher orchestrates the transmission, decoding, and restoration of compressed KV caches in an efficient pipelined manner, eliminating resource contention, masking network fluctuations, and achieving minimum time-to-first-token (TTFT). We prototype KVCodec on diverse GPUs from high- to low-end. Experiments reveal that it reduces TTFT by up to 3.51 times while maintaining lossless accuracy, compared to SOTA methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 57 of its 87 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

57 of 87 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Models3 Oct
  2. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching21 Sep
  3. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  4. 2024Fast State Restoration in LLM Serving with HCache7 Oct
  5. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management1 Oct
  6. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  9. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving4 Mar
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  11. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  12. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
2 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 43

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
31 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  7. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  8. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  9. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  10. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  11. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  12. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  13. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  14. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  15. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  16. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  17. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  18. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  19. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  20. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  21. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  22. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  23. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  24. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  25. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  26. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  27. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  28. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  29. 2025Locality-aware Fair Scheduling in LLM Servingcited by 15
  30. 2025fabric-lib: RDMA Point-to-Point Communication for LLM Systemscited by 8
  31. 2025Self-Taught Agentic Long Context Understandingcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  2. 2025FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference
  3. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
  4. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  5. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  6. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
  7. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  8. 2026The KV Cache Is the New Memory Wall