mlsanityData to 7 Oct 2026

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

arXiv 2606.31145 ↗First posted 30 Jun 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 29 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

22 of 29 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving29 Apr
  2. 2026SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging15 Mar
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  5. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  7. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2024Understanding the planning of LLM agents: A surveycited by 414
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  10. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  11. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  12. 2024Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perceptioncited by 17
1 more
  1. 2025Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategiescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  2. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  3. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching
  4. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  5. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  6. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  7. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  8. 2025Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU