mlsanityData to 7 Oct 2026

Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

arXiv 2607.13205 ↗First posted 14 Jul 2026cs.CL cs.AI cs.LGcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest. On schema-dense input streams such as nested JSON, this score acts as a non-stationary filter that disproportionately retains noise: a non-content sink role (delimiters or whitespace) carries an order of magnitude more energy than any content role, and structural KEY tokens are over-retained at roughly 1.8x the rate of the answer-carrying VALUE tokens, collapsing exact-match accuracy from 88% to 0% at a 5% budget as the signal-to-noise ratio of the retained state degrades. A counterfactual experiment establishes that suppressing KEY tokens is the best deployable filter. Our retraining-free, role-conditional allocation over SnapKV's windowed score, governed by a single tuned hyperparameter, closes 63-98% of the H2O gap at sub-20% budgets and, at higher budgets, modestly matches or exceeds full-cache accuracy -- a small, seed-sensitive denoising effect (borderline significant at B=0.50; not distinguishable from zero at B=0.30 over four seeds). A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains open.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 24 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

24 of 24 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Learning to Evict from Key-Value Cache10 Feb
  2. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference9 Oct
  3. 2025The Pitfalls of KV Cache Compression30 Sep
  4. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation4 Aug
  5. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  6. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference11 Mar
  7. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  8. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  9. 2024When Attention Sink Emerges in Language Models: An Empirical View14 Oct
  10. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
4 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 8

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024Massive Activations in Large Language Modelscited by 277
  5. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  6. 2025Why do LLMs attend to the first token?cited by 122
  7. 2025What are you sinking? A geometric approach on attention sinkcited by 26
  8. 2026SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
  2. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  3. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
  4. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  5. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  6. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  7. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  8. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection