mlsanityData to 7 Oct 2026

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

arXiv 2608.23962 ↗First posted 25 Aug 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache across two, four, or eight devices, buying memory headroom at the price of an all-reduce on every layer and a hardware bill that grows with the device count. The algorithms community shrinks the cache in place, with KV quantisation and eviction keeping a single GPU and spending a little quality instead. Compression papers report memory ratios, parallel-scaling papers report throughput curves, and almost nobody puts the two on the same cost axis. We place tensor-parallel configurations (degree 1 to 8) and KV-compressed configurations (16/8/4-bit, keep-ratios down to 0.25) on one costnormalised axis, cost per million tokens against latency, using a profiled simulator calibrated on A100, A40, and H100 hardware, and we go looking for the cost-equivalence crossover. We do not find one. Across two models (Llama-2 at 7B and 70B), three GPU types, and every level of memory relief we could construct, compression is cheaper by 1.20x to 2.00x. A 7B model on an 80 GB device cannot exhaust its KV budget within its own context window, and the boundary that decides between the strategies is model size relative to device memory, at roughly 36B parameters for an 80 GB card. Below that wall, compression dominates and extra GPUs are largely wasted spend; above it, tensor parallelism stops being a choice and becomes an entry ticket: Llama-2-70B is infeasible on one A100 at any KV setting, because the binding resource is weights, which KV compression does not touch. Tensor parallelism is the only lever that improves latency (compression makes per-token latency worse, by 8 to 93%, through batching contention), while compression is the only lever that multiplies capacity per dollar (16.5x, against 1.21x for an eightfold spend on GPUs).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 24 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

22 of 24 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving16 Dec
  2. 2025KV Cache Transform Coding for Compact Storage in LLM Inference3 Nov
  3. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving31 Mar
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  12. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
4 more
  1. 2019MLPerf Inference Benchmarkcited by 195
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  3. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  4. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The KV Cache Is the New Memory Wall
  2. 2026Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
  3. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  4. 2026MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
  5. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
  6. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  7. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
  8. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs