mlsanityData to 7 Oct 2026

Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression

arXiv 2603.20616 ↗First posted 21 Mar 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-value (KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length, limiting long-context deployment. Existing token eviction methods reduce memory by discarding less important tokens, which can be viewed as a coarse form of dimensionality reduction that assigns each token either zero or full dimension. We propose MixedDimKV, a mixed-dimension KV cache compression method that allocates dimensions to tokens at a more granular level, and MixedDimKV-H, which further integrates head-level importance information. Experiments on long-context benchmarks show that MixedDimKV outperforms prior KV cache compression methods that do not rely on head-level importance profiling. When equipped with the same head-level importance information, MixedDimKV-H consistently outperforms HeadKV. Notably, our approach achieves comparable performance to full attention on LongBench with only 6.25% of the KV cache. Furthermore, in the Needle-in-a-Haystack test, our solution maintains 100% accuracy at a 50K context length while using as little as 0.26% of the cache.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 31 references that are in the map. First posted on arXiv 21 Mar 2026.

Builds on

24 of 31 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  5. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 14

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  11. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  12. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
2 more
  1. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  2. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Accelerationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
  2. 2026The risk of KV cache compression
  3. 2026How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers
  4. 2026Sub-Token Routing for KV Cache Compression
  5. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing
  6. 2026Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit
  7. 2025Inference-Time Hyper-Scaling with KV Cache Compression
  8. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression