mlsanityData to 14 Oct 2026

On the Limits of Learned Importance Scoring for KV Cache Compression

arXiv 2601.14279 ↗First posted 13 Jan 2026cs.LG cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We investigate learned KV cache compression through Speculative Importance Prediction (SIP), a 1.7M parameter non-query-aware scorer that predicts token importance from KV representations alone. Despite architectural sophistication (multi-horizon lookahead, cross-attention), SIP does not outperform simple baselines, including random selection, across 5 seeds, 4 retention levels, and 3 tasks. Key findings: (1) position-based heuristics (keep first 4 + last N tokens) match or exceed learned approaches; (2) prefill attention provides equivalent signal to complex learned scorers; (3) marginal information in KV representations beyond position and prefill attention appears limited for importance prediction. We hypothesize that circular dependence between future queries and generation trajectories contributes to this difficulty.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 28 references that are in the map. First posted on arXiv 13 Jan 2026.

Builds on

19 of 28 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  5. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  6. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 9

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  5. 2022Efficiently Scaling Transformer Inferencecited by 539
  6. 2020A Metric Learning Reality Checkcited by 321
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  8. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  9. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Contentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
  2. 2026Information-Aware KV Cache Compression for Long Reasoning
  3. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction
  4. 2026Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
  5. 2026KV2: A Self-Refining KV Cache
  6. 2026Behavior-Preserving KV Cache Compression
  7. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
  8. 2026KVpop -- Key-Value Cache Compression with Predictive Online Pruning