mlsanityData to 7 Oct 2026

KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing

arXiv 2410.18517 ↗First posted 24 Oct 2024cs.LG cs.AI cs.CLcited by 38

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called textitKVSharer, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that textitKVSharer can reduce KV cache computation by 30%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that textitKVSharer is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 41 of its 45 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

41 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023Mistral 7Bcited by 3,065
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
28 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  10. 2024InternLM2 Technical Reportcited by 391
  11. 2015Skip-Thought Vectorscited by 321
  12. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  13. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  14. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  15. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  16. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  17. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  18. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  19. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  20. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  21. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  22. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  23. 2019Sharing Attention Weights for Fast Transformercited by 34
  24. 2024SubGen: Token Generation in Sublinear Time and Memorycited by 21
  25. 2021LazyFormer: Self Attention with Lazy Updatecited by 16
  26. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  27. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  28. 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 21; 2026: 16.

38 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  4. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  5. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  6. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  7. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  8. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 8
  9. 2025DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compressioncited by 7
  10. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  11. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  12. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
26 more
  1. 2025SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Modelscited by 5
  2. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  3. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inferencecited by 4
  4. 2025Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Evictioncited by 4
  5. 2025XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compressioncited by 4
  6. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  7. 2026The Topological Trouble With Transformerscited by 3
  8. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 3
  9. 2026Search Your Block Floating Point Scales!cited by 2
  10. 2026GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstructioncited by 2
  11. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  12. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharingcited by 1
  13. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  14. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptationcited by 1
  15. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  16. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inferencecited by 0
  17. 2025CIFLEX: Contextual Instruction Flow for Sub-task Execution in Multi-Turn Interactions with a Single On-Device LLMcited by 0
  18. 2026CrossVLA: Cross-Paradigm Post-Training and Inference Optimization for Vision-Language-Action Modelscited by 0
  19. 2026Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inferencecited by 0
  20. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  21. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Servingcited by 0
  22. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  23. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 0
  24. 2026On the Efficiency-Safety Dilemma in Large Reasoning Modelscited by 0
  25. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0
  26. 2026VFold: Symmetry-Aware Cross-Layer Value Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
  2. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2026HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
  7. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  8. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression