mlsanityData to 14 Oct 2026

EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

arXiv 2509.17396 ↗First posted 22 Sep 2025cs.CLcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model's memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near full-cache accuracy under 4-6x compression, and reduces latency and peak memory by up to 2.4x and 3.7x, respectively.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 53 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

36 of 53 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  2. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management21 May
  3. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  4. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  11. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  12. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
15 more
  1. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  2. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  3. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  4. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  5. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  6. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  7. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  8. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  9. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
  10. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 105
  11. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  12. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  13. 2025REALTALK: A 21-Day Real-World Dataset for Long-Term Conversationcited by 35
  14. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  15. 2014Topic Segmentation and Labeling in Asynchronous Conversationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 136
  2. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  3. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 16
  4. 2026Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KVcited by 2
  5. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  6. 2026KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Mergingcited by 0
  7. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  3. 2025CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing
  4. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
  5. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  6. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
  7. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  8. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models