mlsanityData to 7 Oct 2026

KV cache optimization

708 papers7 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: cache, attention, memory, compression, inference

Key facts. KV cache optimization: 708 papers on arXiv since 2022 (data through 7 Oct 2026). 319 of its 708 mapped papers (45%) cite “Efficient Memory Management for Large Language Model Serving with PagedAttention” (2023, arXiv 2309.06180). 313 of its 708 mapped papers (44%) cite “SnapKV: LLM Knows What You are Looking for Before Generation” (2024, arXiv 2404.14469). 45 new papers 24 Sep–7 Oct.

What’s new

24 Sep – 7 Oct 2026

45 new papers, 24 Sep – 7 Oct; 7 this week.

  1. This week · 7
  2. A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
  3. SPIN: Shadow Predictive Indexer for Sparse Attention
  4. Enabling Dynamic Computation in Looped LMs also fits: Serving systems and request scheduling
  5. Monte Carlo Estimation for KV Cache Eviction
  6. OVAL: Output-Aware Local Page Bases for KV Cache Retrieval
  7. Behavior-Preserving KV Cache Compression
  8. DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  9. Earlier · 38
  10. Mask-Guided KV Cache Eviction in Block Diffusion Language Models also fits: Diffusion LLM and MoE inference
  11. AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
  12. MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding
All 45
  1. KV2: A Self-Refining KV Cache
  2. Page-EntroKV: Hardware-Aligned, Entropy-Weighted KV-Cache Eviction under Grouped-Query Attention
  3. SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention
  4. iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD
  5. WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds
  6. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning also fits: Prompt and context compression
  7. EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction
  8. PatchKV: Weight-Space Compensation of KV Cache
  9. SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs
  10. KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
  11. ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
  12. ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation
  13. CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  14. Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
  15. Cartridges++: KV Cache Compression without Off-Context Derailment
  16. CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion
  17. TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash also fits: Serving systems and request scheduling
  18. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding also fits: Serving systems and request scheduling
  19. Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference
  20. Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
  21. Positions Are Not Facts: The Mismatch Between KV Caches and Memory
  22. RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse
  23. FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward
  24. When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
  25. SketchSSM: Write to the Full State, Read from a Compact Sketch
  26. UniCache: Task- and Type-Aware KV Cache Compression for Unified Multimodal Models
  27. Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference
  28. KV-Lingo: Learning KV-Cache Translators with Distillation
  29. Memory as a cache: Exact context reuse and deletion by construction
  30. Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression
  31. Evaluating the accuracy of KV cache reuse techniques
  32. CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters
  33. The KV Cache Is the New Memory Wall also fits: Serving systems and request scheduling
  34. Beyond Mean Attention: Diversity-Aware, Layer-Wise Scoring for KV Cache Eviction
  35. MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression also fits: Prompt and context compression

Foundations

Most cited by papers here

  1. Efficient Memory Management for Large Language Model Serving with PagedAttention2023cited by 319 here
  2. SnapKV: LLM Knows What You are Looking for Before Generation2024cited by 313 here
  3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models2023cited by 290 here
  4. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time2023cited by 182 here
  5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference2024cited by 180 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 290 here
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 172 here
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 194 here
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 209 here
  5. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 59 here
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 72 here
18 more it builds on, not in this chain
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 182 here
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 319 here
  3. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 70 here
  4. 2024Transformers are Multi-State RNNscited by 54 here
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 77 here
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 59 here
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 60 here
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 55 here
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 313 here
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 83 here
  11. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 176 here
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 180 here
  13. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 99 here
  14. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 119 here
  15. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 54 here
  16. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 60 here
  17. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 98 here
  18. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 71 here

The 24 most cited of 716.

Browse

30 of 708, ranked 6–35

  1. 6PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling2024176
  2. 7Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs2023172
  3. 8Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference2024119
  4. 9InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management202499
  5. 10DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads202498
  6. 11PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference202483
  7. 12CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion202472
  8. 13ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference202471
  9. 14CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving202370
  10. 15Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference202460
All 30
  1. 16Palu: Compressing KV-Cache with Low-Rank Projection202460
  2. 17GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM202459
  3. 18Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention202455
  4. 19RazorAttention: Efficient KV Cache Compression Through Retrieval Heads202454
  5. 20Reducing Transformer Key-Value Cache Size with Cross-Layer Attention202451
  6. 21MiniCache: KV Cache Compression in Depth Dimension for Large Language Models202450
  7. 22Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks202450
  8. 23A Survey on Large Language Model Acceleration based on KV Cache Management202450
  9. 24SparQ Attention: Bandwidth-Efficient LLM Inference202350
  10. 25Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning202444
  11. 26KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction202540
  12. 27No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization202440
  13. 28A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression202439
  14. 29LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference202538
  15. 30Eigen Attention: Attention in Low-Rank Space for KV Cache Compression202437
  16. 31FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving202536
  17. 32Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference202435
  18. 33Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption202435
  19. 34Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution202534
  20. 35ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition202434