mlsanityData to 7 Oct 2026

Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving

arXiv 2606.06302 ↗First posted 4 Jun 2026cs.LG cs.SEcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput. Non-uniform KV compression, which allocates heterogeneous budgets across attention heads, preserves accuracy far better than uniform schemes, yet remains impractical: modern serving stacks assume identical KV lengths across heads, so heterogeneity traps freed memory as page fragmentation, spends up to 25% of prefill time reclaiming scattered pages, and skews GPU workloads that inflate decode latency by up to 1.7× or burn 15--20% of each decode step on re-planning. We observe that this heterogeneity need not be discovered at runtime: head-wise retention follows a two-level structural regularity -- an input-invariant head ranking with narrowly bounded per-head ratios -- that can be calibrated offline from as few as 50 samples. Building on this insight, we present Tangram, a serving framework that statically resolves what prior systems handle dynamically: Budget Reservation fixes each head's post-compression footprint at scheduling time, eliminating page reclamation; Ragged Paging clusters similar-budget heads into independent page tables, turning fragmentation into reclaimable memory; and Ahead-of-Time Load Balancing precomputes balanced GPU partitions with zero runtime planning. Implemented on vLLM, Tangram serves as a drop-in substrate for existing non-uniform compression methods, matching their accuracy while improving end-to-end throughput by up to 2.6× over the full-KV baseline. Our implementation is publicly available at https://github.com/aiha-lab/TANGRAM.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 44 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

28 of 44 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction25 Jan
  2. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  3. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  4. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  5. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs2 Mar
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  7. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  8. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  9. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  10. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
1 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  7. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  8. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  11. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  12. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
3 more
  1. 2025Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Modelscited by 58
  2. 2025REALTALK: A 21-Day Real-World Dataset for Long-Term Conversationcited by 31
  3. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  2. 2026SparseEngine: Sparse-First Inference Enginecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
  2. 2026SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Caching
  3. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
  4. 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
  5. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  6. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  7. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  8. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption