mlsanityData to 7 Oct 2026

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs

arXiv 2503.16163 ↗First posted 20 Mar 2025cs.CLcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) have already achieved remarkable results on long-text tasks, but the limited GPU memory (VRAM) resources struggle to accommodate the linearly growing demand for key-value (KV) cache as the sequence length increases, which has become a bottleneck for the application of LLMs on long sequences. Existing KV cache compression methods include eviction, merging, or quantization of the KV cache to reduce its size. However, compression results in irreversible information forgetting, potentially affecting the accuracy of subsequent decoding. In this paper, we propose SpeCache, which takes full advantage of the large and easily expandable CPU memory to offload the complete KV cache, and dynamically fetches KV pairs back in each decoding step based on their importance measured by low-bit KV cache copy in VRAM. To avoid inference latency caused by CPU-GPU communication, SpeCache speculatively predicts the KV pairs that the next token might attend to, allowing us to prefetch them before the next decoding step which enables parallelization of prefetching and computation. Experiments on LongBench and Needle-in-a-Haystack benchmarks verify that SpeCache effectively reduces VRAM usage while avoiding information forgetting for long sequences without re-training, even with a 10x high KV cache compression ratio.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 30 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

27 of 30 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  2. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  5. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 18

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
6 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  4. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  5. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  6. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  2. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
  3. 2025Efficient Low Rank Attention for Long-Context Inference in Large Language Modelscited by 4
  4. 2026Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compressioncited by 2
  5. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inferencecited by 1
  6. 2026Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modescited by 0
  7. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  8. 2026DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inferencecited by 0
  9. 2026ARM: Attention with Routed-Memory for Learnable Sparse Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  3. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  4. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  5. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  6. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
  7. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference