Data to 14 Oct 2026

EPIC: Efficient Position-Independent Caching for Serving Large Language Models

arXiv 2410.15332 ↗First posted 20 Oct 2024cs.LG cs.CL cs.DCcited by 47

Not on the map. Why?

Cited by 29 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

24 of 35 references.

References · 24

2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2020Longformer: The Long-Document Transformercited by 3,413
2023Mistral 7Bcited by 3,164
2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
2024Yi: Open Foundation Models by 01.AIcited by 719
2023In-Context Retrieval-Augmented Language Modelscited by 702
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501

12 more

2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
2024A Survey on Efficient Inference for Large Language Modelscited by 205
2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
2022A Survey on Retrieval-Augmented Text Generationcited by 158
2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
2023Stateful Large Language Model Serving with Pensievecited by 61
2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 40.

47 papers.

Sort:

By area:

2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16
2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generationcited by 13
2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reusecited by 8
2025VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inferencecited by 7
2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 7
2026You Need an Encoder for Native Position-Independent Cachingcited by 6
2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6
2026Models Take Notes at Prefill: KV Cache Can Be Editable and Composablecited by 6
2026KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planningcited by 5

35 more

2026AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agentscited by 4
2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inferencecited by 4
2026InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Contextcited by 3
2026Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Servingcited by 3
2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
2026LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMscited by 3
2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
2026LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encodingcited by 2
2026MiniPIC: Flexible Position-Independent Caching in <100LOCcited by 2
2026HijackKV: New Threat in Position-Independent KV Cache Reusecited by 2
2026Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KVcited by 2
2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inferencecited by 1
2026Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?cited by 1
2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
2026SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariancecited by 1
2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
2026Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMscited by 1
2026What It Costs to Compose, Rebuild, and Correct Precomputed Memorycited by 1
2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 1
2026Context Language Modelscited by 1
2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
2026Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agentscited by 0
2026MaSRead: Content-Addressed Reading of Replicated Latent Storescited by 0
2026Affix Cache for Diffusion Large Language Modelscited by 0
2026RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Servingcited by 0
2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?cited by 0
2026Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMscited by 0
2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reusecited by 0
2026CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusioncited by 0
2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0
2026Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agentscited by 0
2026LatentRemedy: Efficient Remedy For Non-Prefix KV Cache Reuse in Multi-Agent Systemscited by 0
2026Hybrid Reasoning Agentscited by 0

Similar, not cited

Close in wording, no citation either way