mlsanityData to 7 Oct 2026

Monte Carlo Estimation for KV Cache Eviction

arXiv 2610.07643 ↗First posted 6 Oct 2026cs.CL cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter while answering? Since decoding queries are unavailable at eviction time, prior future-aware methods rely on pseudo-responses or synthetic future-query estimates. We cast fixed-budget future-aware eviction as distributional estimation over plausible model-conditional query trajectories and introduce LORE-KV (Lookahead Output-perturbation with Reliability-weighted Ensembles for Key-Value caches), a training-free method that samples short autoregressive continuations from the frozen target model and uses their response-side query states to estimate prompt-token utility. Tokens are scored by projected leave-one-out attention-output deletion cost and aggregated across sampled futures with optional trajectory weighting. The temporary continuations are discarded before final decoding, requiring no auxiliary model or training. Ablations isolate the mechanism: at B=128, a single response-side continuation recovers about 89% of the gain over the prompt-window control, while additional futures provide smaller improvements. At B=128, LORE-KV raises the LongBench average on Qwen2.5-14B from 45.49 to 48.24 (+2.75) and the 16K RULER average on Mistral-7B from 45.20 to 51.05 (+5.85). Gains diminish at larger cache budgets and coexist with task-level regressions. LORE-KV incurs 1.46-2.77x AnDPro's per-sample wall-clock time as a one-time compression overhead across six dense and hybrid-attention backbones.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 61 of its 63 references that are in the map. First posted on arXiv 6 Oct 2026.

Builds on

61 of 63 references.

Closest ancestors · 45

Same direction, newest first

  1. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference25 May
  2. 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM24 May
  3. 2026Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction18 May
  4. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving13 May
  5. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction10 May
  6. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference8 May
  7. 2026Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries12 Mar
  8. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation11 Mar
  9. 2026Learning to Evict from Key-Value Cache10 Feb
  10. 2026Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction9 Feb
  11. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction25 Jan
  12. 2025KV Cache Transform Coding for Compact Storage in LLM Inference3 Nov
33 more
  1. 2025Taming the Fragility of KV Cache Eviction in LLM Inference15 Oct
  2. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference9 Oct
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  4. 2025The Pitfalls of KV Cache Compression30 Sep
  5. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation11 Sep
  6. 2025Adaptive KV-Cache Compression without Manually Setting Budget3 Sep
  7. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction30 Aug
  8. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  9. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  10. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query24 May
  11. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  12. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction18 Apr
  13. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction24 Mar
  14. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  15. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference11 Mar
  16. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  17. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective6 Feb
  18. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  19. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  20. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity3 Dec
  21. 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache27 Nov
  22. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  23. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  24. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  25. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  26. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  27. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  28. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  29. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  30. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  31. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  32. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  33. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 16

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  6. 2024Transformers are Multi-State RNNscited by 120
  7. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  8. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  9. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  10. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  11. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  12. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoningcited by 19
4 more
  1. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  2. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 17
  3. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  4. 2026LongFlow: Efficient KV Cache Compression for Reasoning Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Probabilistic Interpretation of KV Cache Eviction
  2. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
  3. 2026KVpop -- Key-Value Cache Compression with Predictive Online Pruning
  4. 2026When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
  5. 2026QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
  6. 2026Behavior-Preserving KV Cache Compression
  7. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  8. 2026AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning