mlsanityData to 14 Oct 2026

Layer-Condensed KV Cache for Efficient Inference of Large Language Models

arXiv 2405.10637 ↗First posted 17 May 2024cs.CLcited by 59

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Huge memory consumption has been a major bottleneck for deploying high-throughput large language models in real-world applications. In addition to the large number of parameters, the key-value (KV) cache for the attention mechanism in the transformer architecture consumes a significant amount of memory, especially when the number of layers is large for deep language models. In this paper, we propose a novel method that only computes and caches the KVs of a small number of layers, thus significantly saving memory consumption and improving inference throughput. Our experiments on large language models show that our method achieves up to 26× higher throughput than standard transformers and competitive performance in language modeling and downstream tasks. In addition, our method is orthogonal to existing transformer memory-saving techniques, so it is straightforward to integrate them with our model, achieving further improvement in inference efficiency. Our code is available at https://github.com/whyNLP/LCKV.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 38 references that are in the map. First posted on arXiv 17 May 2024.

Builds on

25 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 21

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
9 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  5. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  6. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  7. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 43
  8. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokenscited by 8
  9. 2023Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 23; 2026: 21.

59 papers.

Sort:

By area:

  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  3. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  6. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  7. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  8. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  9. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 41
  10. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  11. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  12. 2025Parallel Continuous Chain-of-Thought with Jacobi Iterationcited by 25
38 more
  1. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 19
  2. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inferencecited by 17
  3. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 17
  4. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  5. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 16
  6. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  7. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reusecited by 15
  8. 2024KV Prediction for Improved Time to First Tokencited by 11
  9. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 9
  10. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  11. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  12. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 6
  13. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  14. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  15. 2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systemscited by 5
  16. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapsecited by 5
  17. 2024Inference-Friendly Models With MixAttentioncited by 4
  18. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  19. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  20. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  21. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  22. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 4
  23. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMscited by 4
  24. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Modelscited by 3
  25. 2024Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Modelscited by 2
  26. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overviewcited by 2
  27. 2026YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inferencecited by 2
  28. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1
  29. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  30. 2025AI Compute Architecture and Evolution Trendscited by 1
  31. 2026Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibilitycited by 1
  32. 2026Do Value Vectors in Deep Layers Need Context from the Residual Stream?cited by 1
  33. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 1
  34. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inferencecited by 0
  35. 2025ShishuLM : Achieving Optimal and Efficient Parameterization with Low Attention Transformer Modelscited by 0
  36. 2025CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperationcited by 0
  37. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMscited by 0
  38. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Method for Building Large Language Models with Predefined KV Cache Capacity
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inference
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  5. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  6. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  7. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
  8. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference