# EPIC: Efficient Position-Independent Caching for Serving Large Language Models

arXiv: https://arxiv.org/abs/2410.15332 · HTML: https://mlsanity.com/paper/2410.15332/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://mlsanity.com/paper/2009.08553/) (2020)
- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.12457/) (2024)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference](https://mlsanity.com/paper/2401.11240/) (2024)

## Built on by (most cited first, 30 of 47)

- [KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse](https://mlsanity.com/paper/2503.16525/) (2025)
- [TraceLab: Characterizing Coding Agent Workloads for LLM Serving](https://mlsanity.com/paper/2606.30560/) (2026)
- [ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.02579/) (2026)
- [MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving](https://mlsanity.com/paper/2502.01960/) (2025)
- [Memory in Large Language Models: Mechanisms, Evaluation and Evolution](https://mlsanity.com/paper/2509.18868/) (2025)
- [RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse](https://mlsanity.com/paper/2603.13289/) (2026)
- [VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference](https://mlsanity.com/paper/2512.12977/) (2025)
- [RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention](https://mlsanity.com/paper/2606.06256/) (2026)
- [You Need an Encoder for Native Position-Independent Caching](https://mlsanity.com/paper/2602.01519/) (2026)
- [KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing](https://mlsanity.com/paper/2606.17034/) (2026)
- [Models Take Notes at Prefill: KV Cache Can Be Editable and Composable](https://mlsanity.com/paper/2606.17107/) (2026)
- [KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning](https://mlsanity.com/paper/2602.23592/) (2026)
- [AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agents](https://mlsanity.com/paper/2604.24039/) (2026)
- [C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference](https://mlsanity.com/paper/2607.17715/) (2026)
- [InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context](https://mlsanity.com/paper/2603.05353/) (2026)
- [Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving](https://mlsanity.com/paper/2605.05696/) (2026)
- [Adaptive KV Cache Reuse for Fast Long-Context LLM Serving](https://mlsanity.com/paper/2605.24022/) (2026)
- [Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse](https://mlsanity.com/paper/2606.23581/) (2026)
- [LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs](https://mlsanity.com/paper/2608.11231/) (2026)
- [Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications](https://mlsanity.com/paper/2601.08833/) (2025)
- [LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding](https://mlsanity.com/paper/2606.04302/) (2026)
- [MiniPIC: Flexible Position-Independent Caching in <100LOC](https://mlsanity.com/paper/2606.13126/) (2026)
- [HijackKV: New Threat in Position-Independent KV Cache Reuse](https://mlsanity.com/paper/2607.19957/) (2026)
- [Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV](https://mlsanity.com/paper/2607.23693/) (2026)
- [MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference](https://mlsanity.com/paper/2511.06010/) (2025)
- [Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?](https://mlsanity.com/paper/2605.27494/) (2026)
- [Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics](https://mlsanity.com/paper/2606.01502/) (2026)
- [SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance](https://mlsanity.com/paper/2606.09441/) (2026)
- [CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals](https://mlsanity.com/paper/2607.28263/) (2026)
- [Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs](https://mlsanity.com/paper/2608.30310/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss.