mlsanityData to 14 Oct 2026

KV Cache Transform Coding for Compact Storage in LLM Inference

arXiv 2511.01815 ↗First posted 3 Nov 2025cs.CL cs.AI cs.LGcited by 13

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20× compression while maintaining reasoning and long-context accuracy, and 40× or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks including AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER. It consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 64 of its 100 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

64 of 100 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  4. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  5. 2024Do Large Language Models Need a Content Delivery Network?16 Sep
  6. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  7. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  10. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
1 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 51

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  11. 2023StarCoder: may the source be with you!cited by 799
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  13. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  14. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  15. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  16. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  17. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  18. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  19. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  20. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  21. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  22. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  23. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  24. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  25. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  26. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  27. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  28. 2023Microscaling Data Formats for Deep Learningcited by 152
  29. 2024Transformers are Multi-State RNNscited by 120
  30. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  31. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  32. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  33. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 67
  34. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  35. 2025Security Concerns for Large Language Models: A Surveycited by 29
  36. 2020Transform Quantization for CNN (Convolutional Neural Network) Compressioncited by 20
  37. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  38. 2019CAT: Compression-Aware Training for bandwidth reductioncited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  2. 2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inferencecited by 3
  3. 2026When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agentscited by 1
  4. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  5. 2026Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantizationcited by 0
  6. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attentioncited by 0
  7. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Servingcited by 0
  8. 2026Low-Bit Recurrent States in Hybrid Language Modelscited by 0
  9. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0
  10. 2026Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compressioncited by 0
  11. 2026FORGE: Procedural Weight Generation for Sparse MoE Inferencecited by 0
  12. 2026AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Codingcited by 0
1 more
  1. 2026Joint Subspace Sparse Attention for Efficient Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  3. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  4. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  5. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
  6. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  7. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  8. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens