mlsanityData to 7 Oct 2026

Efficient LLM Inference with Kcache

arXiv 2404.18057 ↗First posted 28 Apr 2024cs.CLcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models(LLMs) have had a profound impact on AI applications, particularly in the domains of long-text comprehension and generation. KV Cache technology is one of the most widely used techniques in the industry. It ensures efficient sequence generation by caching previously computed KV states. However, it also introduces significant memory overhead. We discovered that KV Cache is not necessary and proposed a novel KCache technique to alleviate the memory bottleneck issue during the LLMs inference process. KCache can be used directly for inference without any training process, Our evaluations show that KCache improves the throughput of popular LLMs by 40% with the baseline, while keeping accuracy.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 12 references that are in the map. First posted on arXiv 28 Apr 2024.

Builds on

10 of 12 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  7. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  8. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Cognitive Memory in Large Language Modelscited by 28
  2. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  8. 2025Online Scheduling for LLM Inference with KV Cache Constraints