mlsanityData to 14 Oct 2026

LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference

arXiv 2511.14510 ↗First posted 18 Nov 2025cs.LGcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and reduce transfers using top-k attention. But their CPU-centric management of the on-GPU cache and CPU-GPU data movement incurs high overhead and fragments the bulk GPU execution that CUDA Graph relies on. To close this gap, we observe that adjacent queries within the same attention head exhibit strong directional similarity and retrieve highly overlapping top-k KV states. This insight enables a simple head granularity cache algorithm, QSAC, in which each head reuses its previously cached KV states whenever the current query is sufficiently similar to the prior one. QSAC further simplifies cache management primitives and cuts CPU involvement almost entirely. We develop LiteCache, a KVCache subsystem that incorporates QSAC. LiteCache introduces a GPU-centric synchronization controller and speculative sparse prefetching, enabling fully overlapped data movement and computation. These mechanisms produce a stable and predictable execution pattern that remains compatible with the bulk execution mode required by CUDA Graphs. Evaluation on two widely-used LLMs indicates that LiteCache achieves comparable accuracy to baselines, while sharply minimizing CPU overhead, fully utilizing PCIe bandwidth, thus improving decoding throughput by 10.7-224.2% on both H100 and A40 GPUs and easily supporting sequence lengths beyond 1M. We opensource LiteCache at https://anonymous.4open.science/r/LiteCache-888D.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 48 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

28 of 48 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference3 Jun
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  9. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2024A Survey on Large Language Models for Code Generationcited by 658
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  11. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  12. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
6 more
  1. 2024Large Language Models Meet NLP: A Surveycited by 103
  2. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  3. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  4. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  5. 2021Memory-efficient Transformers via Top-k Attentioncited by 60
  6. 2013Optimizing CUDA Code By Kernel Fusion---Application on BLAScited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  2. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  5. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
  6. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
  7. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  8. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization