mlsanityData to 7 Oct 2026

MEPIC: Memory Efficient Position Independent Caching for LLM Serving

arXiv 2512.16822 ↗First posted 18 Dec 2025cs.LGcited by 10

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Modern LLM applications such as deep-research assistants, coding agents, and Retrieval-Augmented Generation (RAG) systems, repeatedly process long prompt histories containing shared document or code chunks, creating significant pressure on the Key Value (KV) cache, which must operate within limited memory while sustaining high throughput and low latency. Prefix caching partially alleviates some of these costs by reusing KV cache for previously processed tokens, but limited by strict prefix matching. Position-independent caching (PIC) enables chunk-level reuse at arbitrary positions, but requires selective recomputation and positional-encoding (PE) adjustments. However, because these operations vary across queries, KV for the same chunk diverges across requests. Moreover, without page alignment, chunk KV layouts diverge in memory, preventing page sharing. These issues result in only modest HBM savings even when many requests reuse the same content. We present MEPIC, a memory-efficient PIC system that enables chunk KV reuse across positions, requests, and batches. MEPIC aligns chunk KV to paged storage, shifts recomputation from token- to block-level so only the first block is request-specific, removes positional encodings via Rotary Position Embedding (RoPE) fusion in the attention kernel, and makes remaining blocks fully shareable. These techniques eliminate most duplicate chunk KV in HBM, reducing usage by up to 2x over state-of-the-art PIC at comparable latency and accuracy, and up to 5x for long prompts, without any model changes.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 41 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

27 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving13 Nov
  2. 2025CacheClip: Accelerating RAG with Effective KV Cache Reuse11 Oct

Other references · 25

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  8. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  9. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  10. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  12. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
13 more
  1. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  3. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  4. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  5. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  6. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  7. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  8. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  9. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
  10. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  11. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  12. 2025Generative Caching for Structurally Similar Prompts and Responsescited by 3
  13. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Servingcited by 3
  2. 2026LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMscited by 3
  3. 2026MiniPIC: Flexible Position-Independent Caching in <100LOCcited by 2
  4. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  5. 2026LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encodingcited by 1
  6. 2026HijackKV: New Threat in Position-Independent KV Cache Reusecited by 1
  7. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  8. 2026ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inferencecited by 0
  9. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0
  10. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  2. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
  3. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
  4. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Models
  5. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
  6. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
  7. 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
  8. 2026Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches