mlsanityData to 14 Oct 2026

CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

arXiv 2405.16444 ↗First posted 26 May 2024cs.LGcited by 223

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text's cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? This challenge naturally arises in retrieval-augmented generation (RAG) where the input is supplemented with multiple retrieved texts as the context. We present CacheBlend, a scheme that reuses the precomputed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime, the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job, allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on three open-source LLMs of various sizes and four popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2-3.3x and increases the inference throughput by 2.8-5x from full KV recompute without compromising generation quality. The code is available at https://github.com/LMCache/LMCache.

Key facts. Cited by 223 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 57 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

37 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2023Mistral 7Bcited by 3,164
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
20 more
  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  8. 2024Large Language Models: A Surveycited by 525
  9. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  10. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  12. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  13. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  14. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  15. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  16. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  17. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  18. 2023Retrieval meets Long Context Large Language Modelscited by 115
  19. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  20. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 58; 2026: 160.

223 papers.

Sort:

By area:

  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  2. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 63
  3. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
  4. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  5. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  6. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
  7. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 39
  8. 2024Block-Attention for Efficient Prefillingcited by 37
  9. 2025Cognitive Memory in Large Language Modelscited by 29
  10. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 27
  11. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 26
  12. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
38 more
  1. 2025Auditing Prompt Caching in Language Model APIscited by 20
  2. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 19
  3. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 18
  4. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  5. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
  6. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  7. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  8. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  9. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16
  10. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 13
  11. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scalecited by 13
  12. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generationcited by 13
  13. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  14. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  15. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
  16. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 11
  17. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  18. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  19. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 10
  20. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  21. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  22. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  23. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  24. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  25. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
  26. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  27. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 8
  28. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  29. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reusecited by 8
  30. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  31. 2025VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inferencecited by 7
  32. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 7
  33. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  34. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  35. 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
  36. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6
  37. 2026You Need an Encoder for Native Position-Independent Cachingcited by 6
  38. 2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6

50 most cited of 223 All 223 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
  2. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
  3. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
  4. 2025CacheClip: Accelerating RAG with Effective KV Cache Reuse
  5. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
  6. 2024ThinK: Thinner Key Cache by Query-Driven Pruning
  7. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  8. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs