mlsanityData to 7 Oct 2026

EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving

arXiv 2512.14946 ↗First posted 16 Dec 2025cs.OS cs.AI cs.LGcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity, so prior work has proposed to either evict KV cache to lower-tier storage devices, or compress KV cache so that more KV cache can be fit in the fast memory. However, prior work misses an important opportunity: jointly optimizing the eviction and compression decisions across all KV caches to minimize average generation latency without hurting quality. We propose EVICPRESS, a KV-cache management system that applies lossy compression and adaptive eviction to KV cache across multiple storage tiers. Specifically, for each KV cache of a context, EVICPRESS considers the effect of compression and eviction of the KV cache on the average generation quality and delay across all contexts as a whole. To achieve this, EVICPRESS proposes a unified utility function that quantifies the effect of quality and delay of the lossy compression or eviction. To this end, EVICPRESS's profiling module periodically updates the utility function scores on all possible eviction-compression configurations for all contexts and places KV caches using a fast heuristic to rearrange KV caches on all storage tiers, with the goal of maximizing the utility function scores on each storage tier. Compared to the baselines that evict KV cache or compress KV cache, EVICPRESS achieves higher KV-cache hit rates on fast devices, i.e., lower delay, while preserving high generation quality by applying conservative compression to contexts that are sensitive to compression errors. Evaluation on 12 datasets and 5 models demonstrates that EVICPRESS achieves up to 2.19x faster time-to-first-token (TTFT) at equivalent generation quality.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 56 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

39 of 56 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference14 Nov
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  3. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inference10 Sep
  4. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache30 Aug
  5. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving28 Aug
  6. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  7. 2025Sparse Attention across Multiple-context KV Cache6 Aug
  8. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  9. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  10. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  11. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework11 Mar
  12. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse21 Feb
7 more
  1. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective4 Oct
  2. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  3. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 20

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  10. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  12. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
8 more
  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  2. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  3. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  4. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  5. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  6. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  7. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  8. 2024Adapting Standard Retrieval Benchmarks to Evaluate Generated Answerscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
  2. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
  3. 2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inferencecited by 3
  4. 2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 3
  5. 2026StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Servingcited by 1
  6. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
  7. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Servingcited by 0
  8. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reusecited by 0
  9. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  3. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  4. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  8. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation