mlsanityData to 7 Oct 2026

Probing the Prompt KV Cache: Where It Becomes Dispensable

arXiv 2605.30574 ↗First posted 28 May 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Prior KV cache compression schemes empirically demonstrate that the prompt cache is partially redundant during decoding, dropping or summarising entries with little accuracy loss. We ask when and what kind of redundancy: at which layers, after how many decoding steps, and in what form can the prompt span KV cache be replaced without breaking the task. A controlled splice intervention swept over layer cutoff and decoding steps shows this redundancy is about form (chat template scaffolding) rather than content. Replacing the upper layer prompt span KV cache with KV cache from a chat template scaffold whose user content is a neutral filler recovers near clean accuracy, while zeroing the same slots collapses accuracy. The dissociation replicates across the Qwen3, Gemma 3, and Llama 3 families on multiple datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 14 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

11 of 14 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Whose Narrative is it Anyway? A KV Cache Manipulation Attack16 Nov
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 6

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023In-Context Learning Creates Task Vectorscited by 302
  6. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
  2. 2026Evaluating the accuracy of KV cache reuse techniques
  3. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
  4. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  5. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  6. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  7. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
  8. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration