mlsanityData to 7 Oct 2026

Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression

arXiv 2412.05693 ↗First posted 7 Dec 2024cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Several works have developed eviction policies to remove key-value (KV) pairs from the KV cache for more efficient inference. The focus has been on compressing the KV cache after the input prompt has been processed for faster token generation. In settings with limited GPU memory, and when the input context is longer than the generation length, we show that by also compressing the KV cache during the input processing phase, larger batch sizes can be used resulting in significantly higher throughput while still maintaining the original model's accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 15 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

12 of 15 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  3. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 6

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  4. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  6. 2024Transformers are Multi-State RNNscited by 120

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  3. 2026MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
  4. 2026The KV Cache Is the New Memory Wall
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  8. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review