# Fast KV Compaction via Attention Matching

arXiv: https://arxiv.org/abs/2602.16284 · HTML: https://mlsanity.com/paper/2602.16284/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2026.

## Built on by (most cited first, 30 of 31)

- [Self-Compacting Language Model Agents](https://mlsanity.com/paper/2606.23525/) (2026)
- [Training Transformers for KV Cache Compressibility](https://mlsanity.com/paper/2605.05971/) (2026)
- [Cartridges at Scale: Training Modular KV Caches over Large Document Collections](https://mlsanity.com/paper/2606.04557/) (2026)
- [Parallel Context Compaction for Long-Horizon LLM Agent Serving](https://mlsanity.com/paper/2605.23296/) (2026)
- [Nearly Optimal Attention Coresets](https://mlsanity.com/paper/2605.05602/) (2026)
- [A Simple Plug-in for Improving Eviction-Based KV Cache Compression](https://mlsanity.com/paper/2605.23258/) (2026)
- [End-to-End Context Compression at Scale](https://mlsanity.com/paper/2606.09659/) (2026)
- [Neural Garbage Collection: Learning to Forget while Learning to Reason](https://mlsanity.com/paper/2604.18002/) (2026)
- [Practical Online KV Cache Compaction for LLM Agents: An Empirical Study](https://mlsanity.com/paper/2608.00902/) (2026)
- [Prefix Sliding for efficient test-time scaling](https://mlsanity.com/paper/2608.26070/) (2026)
- [Cartridges++: KV Cache Compression without Off-Context Derailment](https://mlsanity.com/paper/2609.35621/) (2026)
- [Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG](https://mlsanity.com/paper/2603.23562/) (2026)
- [Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction](https://mlsanity.com/paper/2605.06270/) (2026)
- [LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding](https://mlsanity.com/paper/2607.24555/) (2026)
- [ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction](https://mlsanity.com/paper/2609.36835/) (2026)
- [KV-Kaizen: Learning Context-Adaptive Cache Compression Choices](https://mlsanity.com/paper/2609.37988/) (2026)
- [PatchKV: Weight-Space Compensation of KV Cache](https://mlsanity.com/paper/2609.39329/) (2026)
- [Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation](https://mlsanity.com/paper/2604.11628/) (2026)
- [Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA](https://mlsanity.com/paper/2606.32002/) (2026)
- [NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement](https://mlsanity.com/paper/2607.26633/) (2026)
- [Thought-Aware KV Cache Compaction for Reasoning via Adaptive Attention Matching](https://mlsanity.com/paper/2608.12331/) (2026)
- [Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generation](https://mlsanity.com/paper/2608.28276/) (2026)
- [Where Should a Document Live: Context, Representations, or Parameters?](https://mlsanity.com/paper/2609.17346/) (2026)
- [When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression](https://mlsanity.com/paper/2609.33334/) (2026)
- [ReLMem: Learning Recurrent Memory for Longitudinal EHR Modeling](https://mlsanity.com/paper/2609.37587/) (2026)
- [Persistent Context Graphs for Efficient Memory Compaction in LLM Agents](https://mlsanity.com/paper/2609.40118/) (2026)
- [Budgeted Cache Repair for Cross-Context KV-Cache Reuse](https://mlsanity.com/paper/2610.02233/) (2026)
- [The Optimization Landscape of Learning Compacted Context Models](https://mlsanity.com/paper/2610.05885/) (2026)
- [AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD](https://mlsanity.com/paper/2610.06927/) (2026)
- [Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read](https://mlsanity.com/paper/2610.12133/) (2026)

## Abstract (arXiv)

Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.