mlsanityData to 14 Oct 2026

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

arXiv 2603.22910 ↗First posted 24 Mar 2026cs.CLcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 38 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

31 of 38 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction13 Sep
  2. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing22 Aug
  3. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs4 Jun
  4. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query24 May
  5. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression14 Mar
  6. 2025TransMLA: Multi-Head Latent Attention Is All You Need11 Feb
  7. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  8. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  9. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  10. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
1 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
6 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  4. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  5. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  6. 2025Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  2. 2026VFold: Symmetry-Aware Cross-Layer Value Cache Compressioncited by 0
  3. 2026CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Cachescited by 0
  4. 2026BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
  2. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
  3. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  4. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  5. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  6. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
  7. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  8. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration