mlsanityData to 14 Oct 2026

CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences

arXiv 2503.12491 ↗First posted 16 Mar 2025cs.CLcited by 58

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) excel at processing long sequences, boosting demand for key-value (KV) caching. While recent efforts to evict KV cache have alleviated the inference burden, they often fail to allocate resources rationally across layers with different attention patterns. In this paper, we introduce Cascading and Adaptive KV cache Eviction (CAKE), a novel approach that frames KV cache eviction as a "cake-slicing problem." CAKE assesses layer-specific preferences by considering attention dynamics in both spatial and temporal dimensions, allocates rational cache size for layers accordingly, and manages memory constraints in a cascading manner. This approach enables a global view of cache allocation, adaptively distributing resources across diverse attention mechanisms while maintaining memory budgets. CAKE also employs a new eviction indicator that considers the shifting importance of tokens over time, addressing limitations in existing methods that overlook temporal dynamics. Comprehensive experiments on LongBench and NeedleBench show that CAKE maintains model performance with only 3.2% of the KV cache and consistently outperforms current baselines across various models and memory constraints, particularly in low-memory settings. Additionally, CAKE achieves over 10x speedup in decoding latency compared to full cache when processing contexts of 128K tokens with FlashAttention-2. Our code is available at https://github.com/antgroup/cakekv.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 52 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

41 of 52 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  6. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  7. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  8. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2023Mistral 7Bcited by 3,164
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Reformer: The Efficient Transformercited by 1,668
18 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  10. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  11. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  12. 2024Transformers are Multi-State RNNscited by 120
  13. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  14. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  15. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  16. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  17. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  18. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 37.

58 papers.

Sort:

By area:

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  2. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 20
  3. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  4. 2025ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancycited by 17
  5. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  6. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 16
  7. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  8. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  9. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  10. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 11
  11. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 10
  12. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
38 more
  1. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  2. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generationcited by 9
  3. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  4. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cachecited by 7
  5. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compressioncited by 7
  6. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  7. 2026Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Evictioncited by 6
  8. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  9. 2026Why Attention Patterns Exist: A Unifying Temporal Perspective Analysiscited by 5
  10. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 5
  11. 2025Efficient Low Rank Attention for Long-Context Inference in Large Language Modelscited by 4
  12. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 4
  13. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  14. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  15. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  16. 2025Adaptive KV-Cache Compression without Manually Setting Budgetcited by 3
  17. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  18. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
  19. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  20. 2025Autoregressive Image Generation Needs Only a Few Lines of Cached Tokenscited by 2
  21. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 2
  22. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inferencecited by 2
  23. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  24. 2026T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routingcited by 2
  25. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1
  26. 2025Training-free Context-adaptive Attention for Efficient Long Context Modelingcited by 1
  27. 2026Dynamic Thinking-Token Selection for Efficient Reasoning in Large Reasoning Modelscited by 1
  28. 2026Hybrid Associative Memoriescited by 1
  29. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  30. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inferencecited by 1
  31. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  32. 2026Exploring a Layer-Wise Design Space for KV Cache Evictioncited by 1
  33. 2026Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inferencecited by 1
  34. 2026RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagationcited by 1
  35. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 1
  36. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  37. 2026TGV-KV: Text-Grounded KV Eviction for Vision-Language Modelscited by 0
  38. 2026PReM: Learning What to Preserve and When to Refresh for Context Compressioncited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  3. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
  4. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  5. 2026Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
  6. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference