mlsanityData to 7 Oct 2026

A Simple Plug-in for Improving Eviction-Based KV Cache Compression

arXiv 2605.23258 ↗First posted 22 May 2026cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. Experimental results show that VECTOR improves quality-memory trade-offs under medium-to-high compression, with especially clear gains in stricter budget regimes.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 33 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

31 of 33 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs19 Feb
  2. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference8 Feb
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  4. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  5. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  6. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection3 Mar
  7. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models31 Jan
  8. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
1 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2025OpenAI GPT-5 System Cardcited by 910
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
6 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  3. 2024Massive Activations in Large Language Modelscited by 277
  4. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  5. 2026Fast KV Compaction via Attention Matchingcited by 28
  6. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 2
  2. 2026Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Evictioncited by 0
  3. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVDcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
  2. 2026Behavior-Preserving KV Cache Compression
  3. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  4. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  5. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
  6. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  8. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens