mlsanityData to 14 Oct 2026

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

arXiv 2405.12532 ↗First posted 21 May 2024cs.CLcited by 147

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference, hindering their scalability for real-time applications like chatbots. To accelerate inference, we store computed keys and values (KV cache) in the GPU memory. Existing methods study the KV cache compression to reduce memory by pruning the pre-computed KV cache. However, they neglect the inter-layer dependency between layers and huge memory consumption in pre-computation. To explore these deficiencies, we find that the number of crucial keys and values that influence future generations decreases layer by layer and we can extract them by the consistency in attention weights. Based on the findings, we propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer saves significant memory by computing fewer keys and values without sacrificing performance. Experimental results show PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache.

Key facts. Cited by 147 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 29 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

17 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
2 more
  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 216

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 71; 2026: 46.

147 papers.

Sort:

By area:

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  6. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  7. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  8. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  9. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  10. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  11. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 66
  12. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
38 more
  1. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58
  2. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  3. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  4. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  5. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 44
  6. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 43
  7. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 41
  8. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  9. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  10. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  11. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  12. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  13. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 31
  14. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  15. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  16. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 26
  17. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tilecited by 26
  18. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  19. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  20. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  21. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  22. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  23. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  24. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Accelerationcited by 14
  25. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  26. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modelingcited by 13
  27. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  28. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generationcited by 13
  29. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 12
  30. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  31. 2024RefreshKV: Updating Small KV Cache During Long-form Generationcited by 11
  32. 2025GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awarenesscited by 11
  33. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 11
  34. 2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 11
  35. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 10
  36. 2025IAM: Efficient Inference through Attention Mapping between Different-scale LLMscited by 10
  37. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inferencecited by 10
  38. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
  3. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  6. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  7. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  8. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference