mlsanityData to 14 Oct 2026

SGD-KV: Summarization Guided KV Cache Compression

arXiv 2609.03235 ↗First posted 3 Sep 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different attention heads. We present SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation. Experiments on Qwen2.5-7B-1M and Qwen3-32B across diverse long-context benchmarks demonstrate that SGD-KV achieves state-of-the-art performance with contexts up to 1M tokens, while reducing KV cache memory usage by up to 75%. Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 27 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

25 of 27 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  3. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 17

  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  7. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  9. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  10. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  11. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 183
  12. 2025Qwen2.5-1M Technical Reportcited by 164
5 more
  1. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  2. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  3. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  4. 2013Complexity of Word Collocation Networks: A Preliminary Structural Analysiscited by 17
  5. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  4. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
  5. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  6. 2026StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference
  7. 2026Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
  8. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction