mlsanityData to 7 Oct 2026

Divergence Timing and Cumulative Disagreement under KV-Cache Eviction

arXiv 2609.16617 ↗First posted 15 Sep 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV-cache eviction perturbs the conditional token distributions governing autoregressive generation. We investigate how first-divergence timing and subsequent token mismatch determine cumulative disagreement. We derive an exact decomposition under a specified stepwise maximal coupling: the expected mismatch fraction equals a first-mismatch contribution plus post-divergence exposure multiplied by its mismatch rate. An explicit construction over unrestricted autoregressive kernel pairs realizes the sharp interval of risks compatible with a finite divergence-aligned observation window. Residual-branch conditional Monte Carlo provides unbiased joint estimates of occurrence, occupation, and window/tail contributions, with per-replicate variance dominance for total token loss. Complete trajectories from Meta-Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct show that SnapKV at 50% retention enters divergence later and less often than SnapKV-512 or recent-token retention with the same 50% prompt-cache budget, while post-divergence total variation (TV) remains high. In an exploratory analysis of 288 documents, post-divergence exposure accounts for 85-90% of four aggregate mismatch gaps. On 288 independent documents at 90% retention, prespecified comparisons show higher branch-aligned TV in the late than in the early window in both models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 36 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

21 of 36 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Information-Aware KV Cache Compression for Long Reasoning25 Jun
  2. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation11 Mar
  3. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving31 Mar
  4. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  5. 2025Compression Barriers for Autoregressive Transformers21 Feb
  6. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  7. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  8. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  9. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 10

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  7. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  8. 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 33
  9. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 7
  10. 2026Total Variation Distance Estimation in Autoregressive Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference
  2. 2026Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference
  3. 2025Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification
  4. 2025Taming the Fragility of KV Cache Eviction in LLM Inference
  5. 2026Monte Carlo Estimation for KV Cache Eviction
  6. 2026Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens
  7. 2026Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
  8. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models