mlsanityData to 7 Oct 2026

Joint Encoding of KV-Cache Blocks for Scalable LLM Serving

arXiv 2601.03067 ↗First posted 6 Jan 2026cs.LG cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Modern large language models (LLMs) drive interactive AI systems but are bottlenecked by the memory-heavy growth of key-value (KV) caches, which limits real-time throughput under concurrent loads. Existing KV-cache compression methods rely on rigid heuristics, disrupt tensor layouts, or require specialized compute, hindering scalability and deployment. We propose joint encoding of KV-cache blocks, which fuses similar blocks across requests and input chunks into shared representations while preserving standard cache structure. This alleviates the KV-cache memory bottleneck, supporting high-concurrency serving without specialized hardware. Theoretically, we analyze the rate-distortion tradeoff of fused cache blocks under a Poisson process model. Empirically, our method achieves up to 4.38 × KV-cache compression with negligible accuracy loss across diverse LLMs and benchmarks, outperforming recent structured and adaptive compression baselines. In real LLM serving, joint encoding improves the token throughput by ~40% on a single-machine vLLM benchmark, demonstrating substantial gains in inference throughput. Code is available at https://github.com/sef1/kvfastfusion kvjointencoding.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 34 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

19 of 34 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration30 May
  2. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  3. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs13 Mar
  4. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse21 Feb
  5. 2025TransMLA: Multi-Head Latent Attention Is All You Need11 Feb
  6. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 11

  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  7. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  8. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  9. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  10. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  11. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Designcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse
  4. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  5. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2025KV Cache Transform Coding for Compact Storage in LLM Inference
  8. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving