mlsanityData to 7 Oct 2026

Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference

arXiv 2609.32663 ↗First posted 26 Sep 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The memory usage and decoding latency of LLM inference grow rapidly with context length. To reduce these costs, key-value (KV) cache compression methods selectively retain cached states based on token importance or differences in attention patterns across heads. However, we discover that retrieval capability varies substantially with relative distance, even within the same attention head. To exploit this structure, we introduce Distance-KV, which learns a static KV retention pattern over the joint space of layers, attention heads, and relative distances. The pattern is learned offline with the language model frozen and reused across inputs to prune and compact the KV cache without online importance scoring. Across three backbone models and four long-context benchmarks, Distance-KV consistently achieves the best overall performance among competing KV cache compression methods, exceeding the strongest compression baseline by up to 9.3 points on RULER at 128K. On Llama-3.1-8B-Instruct at 128K, Distance-KV reduces KV cache memory by 65.4% and achieves a 1.66× decoding speedup relative to Dense. Together, these results identify relative distance as an important structural dimension for understanding how LLMs retrieve information over long contexts and for designing more efficient inference methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 23 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

21 of 23 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Taming the Fragility of KV Cache Eviction in LLM Inference15 Oct
  2. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  11. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  12. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
1 more
  1. 2026Training-Inference Consistent Segmented Execution for Long-Context LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
  2. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  3. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  4. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  5. 2026HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
  6. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  7. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
  8. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference