mlsanityData to 7 Oct 2026

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

arXiv 2405.14366 ↗First posted 23 May 2024cs.CL cs.AI cs.LGcited by 122

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 66 of its 77 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

66 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  2. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024Mixtral of Expertscited by 1,785
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  14. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  15. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  16. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  17. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  18. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  19. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  20. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  21. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  22. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  23. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  24. 2018CoQA: A Conversational Question Answering Challengecited by 674
  25. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  26. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  27. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  28. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  29. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  30. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  31. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  32. 2022Efficiently Scaling Transformer Inferencecited by 529
  33. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  34. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  35. 2021Random Feature Attentioncited by 350
  36. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  37. 2022Confident Adaptive Language Modelingcited by 297
  38. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 46; 2026: 53.

122 papers.

Sort:

By area:

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  5. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  6. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  9. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  10. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  11. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  12. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
38 more
  1. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  2. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  3. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  4. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  5. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  6. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  7. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  8. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  9. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  10. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  11. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  12. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
  13. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 16
  14. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  15. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  16. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  17. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
  18. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12
  19. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  20. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  21. 2024Cross-Self KV Cache Pruning for Efficient Vision-Language Inferencecited by 11
  22. 2025TokenButler: Token Importance is Predictablecited by 11
  23. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  24. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  25. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMscited by 9
  26. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  27. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  28. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8
  29. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 8
  30. 2024Lossless KV Cache Compression to 2%cited by 7
  31. 2024AdaCM2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reductioncited by 7
  32. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  33. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  34. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  35. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  36. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  37. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  38. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
  4. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  7. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization