mlsanityData to 7 Oct 2026

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

arXiv 2406.02069 ↗First posted 4 Jun 2024cs.CL cs.AIcited by 420

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In this study, we investigate whether attention-based information flow inside large language models (LLMs) is aggregated through noticeable patterns for long context processing. Our observations reveal that LLMs aggregate information through Pyramidal Information Funneling where attention is scattering widely in lower layers, progressively consolidating within specific contexts, and ultimately focusing on critical tokens (a.k.a massive activation or attention sink) in higher layers. Motivated by these insights, we developed PyramidKV, a novel and effective KV cache compression method. This approach dynamically adjusts the KV cache size across different layers, allocating more cache in lower layers and less in higher ones, diverging from traditional methods that maintain a uniform KV cache size. Our experimental evaluations, utilizing the LongBench benchmark, show that PyramidKV matches the performance of models with a full KV cache while retaining only 12% of the KV cache, thus significantly reducing memory usage. In scenarios emphasizing memory efficiency, where only 0.7% of the KV cache is maintained, PyramidKV surpasses other KV cache compression techniques, achieving up to a 20.5 absolute accuracy improvement on TREC dataset. In the Needle-in-a-Haystack experiment, PyramidKV outperforms competing methods in maintaining long-context comprehension in LLMs; notably, retaining just 128 KV cache entries enables the LLAMA-3-70B model to achieve 100.0 Acc. performance.

Key facts. Cited by 420 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 42 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

31 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  3. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Mistral 7Bcited by 3,065
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  11. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 590
14 more
  1. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  2. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  3. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  4. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  5. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  6. 2021Efficient Attentions for Long Document Summarizationcited by 342
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2024Massive Activations in Large Language Modelscited by 277
  9. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  10. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  11. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  12. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  13. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  14. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 135; 2026: 247.

420 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  4. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  7. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  8. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  9. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  10. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  11. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  12. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
38 more
  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  3. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  4. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  5. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  7. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  8. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  9. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  10. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  11. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
  12. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  13. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  14. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  15. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  16. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  17. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  18. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  19. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  20. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  21. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  22. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  23. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  24. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  25. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  26. 2025Cognitive Memory in Large Language Modelscited by 28
  27. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  28. 2025Tensor Product Attention Is All You Needcited by 27
  29. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  30. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  31. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  32. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 24
  33. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  34. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  35. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMscited by 21
  36. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  37. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  38. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20

50 most cited of 420 All 420 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  2. 2026Kwai Summary Attention Technical Report
  3. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  4. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
  5. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  6. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  7. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  8. 2026DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation