mlsanityData to 14 Oct 2026

PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

arXiv 2509.04377 ↗First posted 4 Sep 2025cs.LGcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM's PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 39 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

29 of 39 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  3. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  6. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  11. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  12. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
3 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  11. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  12. 2021Efficient Attentions for Long Document Summarizationcited by 345
2 more
  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  2. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttentioncited by 4
  2. 2026Epiphany-Aware KV Cache Eviction Without the Attention Matrixcited by 2
  3. 2025PatternKV: Flattening KV Representation Expands Quantization Headroomcited by 0
  4. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  5. 2026Discovering KV Cache Eviction Policies via LLM-Guided Program Evolutioncited by 0
  6. 2026GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Servingcited by 0
  7. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ThinK: Thinner Key Cache by Query-Driven Pruning
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  3. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
  4. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  5. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  6. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
  7. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  8. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression