# Do Large Language Models Need a Content Delivery Network?

arXiv: https://arxiv.org/abs/2409.13761 · HTML: https://mlsanity.com/paper/2409.13761/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [RAFT: Adapting Language Model to Domain Specific RAG](https://mlsanity.com/paper/2403.10131/) (2024)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.12457/) (2024)
- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture](https://mlsanity.com/paper/2401.08406/) (2024)
- [Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases](https://mlsanity.com/paper/2403.10446/) (2024)
- [Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue](https://mlsanity.com/paper/2406.06399/) (2024)

## Built on by (most cited first, 16 of 16)

- [PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications](https://mlsanity.com/paper/2505.07203/) (2025)
- [Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2601.06007/) (2026)
- [METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation](https://mlsanity.com/paper/2412.10543/) (2024)
- [KV Cache Transform Coding for Compact Storage in LLM Inference](https://mlsanity.com/paper/2511.01815/) (2025)
- [HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse](https://mlsanity.com/paper/2504.02921/) (2025)
- [Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse](https://mlsanity.com/paper/2511.00413/) (2025)
- [MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference](https://mlsanity.com/paper/2504.09775/) (2025)
- [Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents](https://mlsanity.com/paper/2609.22486/) (2026)
- [PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Serving](https://mlsanity.com/paper/2504.20101/) (2025)
- [Reasoning Language Model Inference Serving Unveiled: An Empirical Study](https://mlsanity.com/paper/2510.18672/) (2025)
- [Revolutionizing Long-Term Memory in AI: New Horizons with High-Capacity and High-Speed Storage](https://mlsanity.com/paper/2602.16192/) (2026)
- [Voice-based AI Agents: Filling the Economic Gaps in Digital Health Delivery](https://mlsanity.com/paper/2507.16229/) (2025)
- [Accelerating Language Model Workflows with Prompt Choreography](https://mlsanity.com/paper/2512.23049/) (2025)
- [VineLM: Trie-Based Fine-Grained Control for Agentic Workflows](https://mlsanity.com/paper/2605.23914/) (2026)
- [Positions Are Not Facts: The Mismatch Between KV Caches and Memory](https://mlsanity.com/paper/2609.33759/) (2026)
- [Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute](https://mlsanity.com/paper/2610.10845/) (2026)

## Abstract (arXiv)

As the use of large language models (LLMs) expands rapidly, so does the range of knowledge needed to supplement various LLM queries. Thus, enabling flexible and efficient injection of new knowledge in LLM inference is critical. Three high-level options exist: (i) embedding the knowledge in LLM's weights (i.e., fine-tuning), (ii) including the knowledge as a part of LLM's text input (i.e., in-context learning), or (iii) injecting the KV caches of the new knowledge to LLM during prefill. This paper argues that, although fine-tuning and in-context learning are popular, using KV caches as the medium of knowledge could simultaneously enable more modular management of knowledge injection and more efficient LLM serving with low cost and fast response. To realize these benefits, we envision a Knowledge Delivery Network (KDN), a new system component in LLM services that dynamically optimizes the storage, transfer, and composition of KV cache across LLM engines and other compute and storage resources. We believe that, just like content delivery networks (CDNs), such as Akamai, enabled the success of the Internet ecosystem through their efficient data delivery, KDNs will be critical to the success of LLM applications through their efficient knowledge delivery. We have open-sourced a KDN prototype at https://github.com/LMCache/LMCache.