mlsanityData to 13 Oct 2026

On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

arXiv 2402.06262 ↗First posted 9 Feb 2024cs.CL cs.AIcited by 32

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands. In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length. As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget. This paper embarks on the efficacy of existing eviction policies in terms of importance score calculation and eviction scope construction. We identify the deficiency of prior policies in these two aspects and introduce RoCo, a robust cache omission policy based on temporal attention scores and robustness measures. Extensive experimentation spanning prefilling and auto-regressive decoding stages validates the superiority of RoCo. Finally, we release EasyKV, a versatile software package dedicated to user-friendly key-value constrained generative inference. Code available at https://github.com/DRSY/EasyKV.

Key facts. Cited by 32 papers in the map (data through 13 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 45 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

32 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 31

  1. 2017Attention Is All You Needcited by 60,106
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  8. 2020Longformer: The Long-Document Transformercited by 3,393
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
19 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  2. 2020Rethinking Attention with Performerscited by 1,594
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,386
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,288
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 896
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
  13. 2023The Falcon Series of Open Language Modelscited by 533
  14. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 409
  15. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 381
  16. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  17. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 101
  18. 2022The Devil in Linear Transformercited by 98
  19. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 17; 2025: 12; 2026: 3.

32 papers.

Sort:

By area:

  1. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 232
  2. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  3. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  4. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 92
  5. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 72
  6. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58
  7. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43
  8. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  9. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  10. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  11. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  12. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
20 more
  1. 2024Towards LifeSpan Cognitive Systemscited by 13
  2. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  3. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  4. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12
  5. 2024Cross-Self KV Cache Pruning for Efficient Vision-Language Inferencecited by 11
  6. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  7. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMscited by 9
  8. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inferencecited by 9
  9. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inferencecited by 7
  10. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertaintycited by 7
  11. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  12. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMscited by 7
  13. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modelingcited by 4
  14. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  15. 2024Finch: Prompt-guided Key-Value Cache Compressioncited by 2
  16. 2025Autoregressive Image Generation Needs Only a Few Lines of Cached Tokenscited by 2
  17. 2026Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLMcited by 2
  18. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  19. 2024Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compressioncited by 0
  20. 2026Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
  2. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
  3. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  4. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  5. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
  6. 2026Learning to Evict from Key-Value Cache
  7. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models