FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
KV cache techniques in Transformer models aim to reduce redundant computations at the expense of substantially increased memory usage, making KV cache compression an important and popular research topic. Recently, state-of-the-art KV cache compression methods implement imbalanced, per-head allocation algorithms that dynamically adjust the KV cache budget for each attention head, achieving excellent performance in single-GPU scenarios. However, we observe that such imbalanced compression leads to significant load imbalance when deploying multi-GPU inference, as some GPUs become overburdened while others remain underutilized. In this paper, we propose FairKV, a method designed to ensure fair memory usage among attention heads in systems employing imbalanced KV cache compression. The core technique of FairKV is Fair-Copying, which replicates a small subset of memory-intensive attention heads across GPUs using data parallelism to mitigate load imbalance. Our experiments on popular models, including LLaMA 70b and Mistral 24b model, demonstrate that FairKV increases throughput by 1.66x compared to standard tensor parallelism inference. Our code will be released as open source upon acceptance.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 21 references that are in the map. First posted on arXiv 19 Feb 2025.
Builds on
17 of 21 references.
Closest ancestors · 8
Same direction, newest first
- 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
- 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
- 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
Other references · 9
- 2017Attention Is All You Needcited by 60,361
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
- 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
- 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
- 2026Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
- 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
- 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
- 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
- 2025PiKV: KV Cache Management System for Mixture of Experts
- 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving