mlsanityData to 7 Oct 2026

CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

arXiv 2310.07240 ↗First posted 11 Oct 2023cs.NI cs.LGcited by 199

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 63 of its 152 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

63 of 152 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  13. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  14. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  15. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  16. 2022Efficiently Scaling Transformer Inferencecited by 529
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  19. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  20. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  21. 2021A Survey of Transformerscited by 399
  22. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  23. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  24. 2022Memorizing Transformerscited by 219
  25. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  26. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  27. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  28. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  29. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  30. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  31. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  32. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  33. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  34. 2016Tensor Ring Decompositioncited by 125
  35. 2018Practical Full Resolution Learned Lossless Image Compressioncited by 93
  36. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  37. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  38. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 75; 2026: 107.

199 papers.

Sort:

By area:

  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  4. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  7. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  9. 2025Efficient Reasoning with Hidden Thinkingcited by 72
  10. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  11. 2023Stateful Large Language Model Serving with Pensievecited by 59
  12. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
38 more
  1. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  3. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  4. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  5. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  6. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
  7. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  8. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  9. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  10. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  11. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  12. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 23
  13. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  14. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  15. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  16. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  17. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 16
  18. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  19. 2025TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generationcited by 15
  20. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  21. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  22. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  23. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  24. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  25. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inferencecited by 10
  26. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
  27. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9
  28. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
  29. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reusecited by 8
  30. 2025Q-KVComm: Efficient Multi-Agent Communication Via Adaptive KV Cache Compressioncited by 8
  31. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  32. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  33. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  34. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  35. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Importantcited by 7
  36. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  37. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  38. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  2. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  6. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference