mlsanityData to 14 Oct 2026

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

arXiv 2608.21362 ↗First posted 21 May 2026cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing prefix-caching systems reduce this cost but require prompts to share a leading contiguous prefix, limiting effectiveness when shared content appears at arbitrary positions. We present KVBoost, a chunk-level KV cache reuse system for HuggingFace-compatible decoder models that enables reuse regardless of content position. KVBoost introduces a dual-hash keying scheme that separates positional identity (prefix hash) from content identity (content hash), supporting both exact and approximate cache matches. To address attention boundary errors from independently cached chunks, KVBoost employs two repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which identifies and recomputes high-deviation tokens after a probe pass. The system further incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget. Evaluated on Qwen/Qwen2.5-3B over 1,000 bug-localization samples, KVBoost achieves a 4.49x reduction in time-to-first-token (142.4 ms vs. 639.1 ms) and outperforms prefix caching by 16%, with no loss in accuracy (99.2% vs. 99.1%). KVBoost provides a practical, memory-bounded inference acceleration layer compatible with RoPE-based models without architectural modification.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 27 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

11 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  2. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 7

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  3. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
  4. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
  5. 2025KVCrush: Key value cache size-reduction using similarity in head-behaviour
  6. 2025H1B-KV: Hybrid One-Bit Caches for Memory-Efficient Large Language Model Inference
  7. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
  8. 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse