mlsanityData to 7 Oct 2026

KV-Distill: Nearly Lossless Learnable Context Compression for LLMs

arXiv 2503.10337 ↗First posted 13 Mar 2025cs.CL cs.AIcited by 19

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Sequence-to-sequence tasks often benefit from long contexts, but the quadratic complexity of self-attention in standard Transformers renders this non-trivial. During generation, temporary representations -stored in the so-called KV cache-account for a large portion of GPU memory usage and scale linearly with context length. We introduce KV-Distill, a Transformer compression framework that distills long context KV caches into significantly shorter representations in a question-independent fashion. KV-Distill can be trained as a parameter-efficient adaptor for pretrained models, and enables the compression of arbitrary spans of a context while preserving pre-trained model capabilities. We treat a compressed-uncompressed cache as a student-teacher pairing and apply a KL-type divergence to match the generated outputs. KV-Distill outperforms other compression techniques in worst-case extractive tasks and approaches uncompressed performance in long context question answering and summarization, and it can be fine-tuned on domain-specific contexts to reduce lengths by up to 99% while preserving downstream performance. We demonstrate the generalizability of KV-Distill across various model sizes and architectures.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 31 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

16 of 31 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  11. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  12. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
4 more
  1. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  2. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  3. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  4. 2023Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokenscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  3. 2025Towards General Continuous Memory for Vision-Language Modelscited by 14
  4. 2026Learning to Evict from Key-Value Cachecited by 12
  5. 2025KaVa: Latent Reasoning via Compressed KV-Cache Distillationcited by 6
  6. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descentcited by 6
  7. 2025CompLLM: Compression for Long Context Q&Acited by 5
  8. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  9. 2026Self-Policy Distillation via Capability-Selective Subspace Projectioncited by 3
  10. 2026Still: Amortized KV Cache Compaction in a Single Forward Passcited by 3
  11. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 2
  12. 2026Low-Rank Key Value Attentioncited by 1
7 more
  1. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  2. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 1
  3. 2026ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compactioncited by 1
  4. 2026Learning What to Remember: Test-Time Training via Context Distillationcited by 0
  5. 2026Sigmoid Attention as a Better Substrate for Learned KV Cache Evictioncited by 0
  6. 2026Compression-Aware Abstention: Teaching LLMs to Refuse When KV-Compression Masks Remove Answer Evidencecited by 0
  7. 2026The Optimization Landscape of Learning Compacted Context Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  2. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  3. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  4. 2026Training Transformers for KV Cache Compressibility
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  6. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  7. 2026End-to-End Context Compression at Scale
  8. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference