mlsanityData to 14 Oct 2026

Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores

arXiv 2507.08143 ↗First posted 10 Jul 2025cs.CL cs.AIcited by 17

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Modern Large Language Models (LLMs) are increasingly trained to support very large context windows. We present Compactor, a training-free, query-agnostic KV compression strategy that uses approximate leverage scores to determine token importance. We show that Compactor can achieve the same performance as competing methods while retaining 20% fewer tokens in both synthetic and real-world context tasks, while being more task-robust. We further introduce a procedure for context-calibrated compression: inferring the maximum compression a given context supports before significant performance loss. Using context-calibrated compression, we show that Compactor achieves full KV performance on Longbench while reducing the KV memory burden by 68%, on average. To demonstrate the efficacy and generalizability of our approach, we apply Compactor to 27 synthetic and real-world tasks from RULER and Longbench, with models from both the Qwen 2.5 and Llama 3.1 families. Finally, we release compactor-vllm, an inference engine and suite of optimized Triton kernels designed to efficiently support the sparse, non-contiguous memory access patterns inherent to compressed KV caches. This work demonstrates that Compactor offers a practical, high-performance solution for alleviating the memory bottleneck in modern LLM deployment.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 47 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

34 of 47 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  2. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  10. 2024KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation8 May
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
10 more
  1. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  2. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  4. 2024Transformers are Multi-State RNNscited by 120
  5. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  6. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  7. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  8. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19
  9. 2021Fast Statistical Leverage Score Approximation in Kernel Ridge Regressioncited by 14
  10. 2023SALSA: Sequential Approximate Leverage-Score Algorithm with Application in Analyzing Big Time Series Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 15
  2. 2026Learning to Evict from Key-Value Cachecited by 14
  3. 2026Multi-Vector Index Compression in Any Modalitycited by 10
  4. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  5. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  6. 2026Still: Amortized KV Cache Compaction in a Single Forward Passcited by 4
  7. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  8. 2026Rethinking LoRA Memory Through the Lens of KV Cache Compressioncited by 2
  9. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  10. 2026ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compactioncited by 1
  11. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  12. 2026KV2: A Self-Refining KV Cachecited by 1
5 more
  1. 2026How Query Visibility Changes KV-Cache Compression Rankings: A Matched-Budget Auditcited by 0
  2. 2026Emergent Structure in the Marginal Attention Space of Language Modelscited by 0
  3. 2026The Optimization Landscape of Learning Compacted Context Modelscited by 0
  4. 2026Generative Adversarial Loopscited by 0
  5. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  4. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  5. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  6. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  7. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
  8. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget