mlsanityData to 14 Oct 2026

KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs

arXiv 2608.15943 ↗First posted 16 Aug 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Pipeline parallelism (PP) is widely used to scale large language model (LLM) training, but its efficiency is often limited by stage imbalance and pipeline bubbles. Meanwhile, cross-layer KV sharing has primarily been studied as a mechanism for reducing KV-cache costs during inference, without examining how KV reuse reshapes pipeline workloads. We present KV-Pipe, a stage-aware KV-sharing mechanism that turns KV reuse into a pipeline-balancing control knob. KV-Pipe starts from the tail stage, converts selected attention layers to cross-layer KV sharing in a tail-first order, and iteratively retargets the current bottleneck to drive the FLOPs Imbalance Ratio (FIR) toward 1. The procedure is performed offline and requires only a pipeline partition and per-layer FLOPs estimates, introducing negligible runtime overhead and requiring no online tuning. Across multiple pipeline-parallel configurations, KV-Pipe consistently improves utilization and throughput, achieving up to 9.2% higher training MFU and up to a 9.8% reduction in iteration time, with larger gains at higher pipeline-parallel degrees where stage imbalance is amplified. Furthermore, the same KV-sharing mechanism provides an inference-side benefit by reducing KV-cache growth and redundant KV projection work, resulting in higher decoding throughput for long-context workloads. These results identify KV layout as a system--architecture degree of freedom for jointly improving pipeline-parallel training efficiency and long-context inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 35 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

26 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference18 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 24

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  8. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  9. 2024Nemotron-4 340B Technical Reportcited by 144
  10. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  11. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  12. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
12 more
  1. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  2. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  3. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
  4. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  5. 2023DynaPipe: Optimizing Multi-task Training through Dynamic Pipelinescited by 12
  6. 2024Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Trainingcited by 11
  7. 2025TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inferencecited by 8
  8. 2025SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Trainingcited by 6
  9. 2024PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Trainingcited by 5
  10. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 5
  11. 2025ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Trainingcited by 3
  12. 2025DawnPiper: A Memory-scablable Pipeline Parallel Training Frameworkcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  4. 2026VPP: Virtual Pipeline Parallelism for Efficient Chunked Prefill in Long-Context LLM Inference
  5. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Serving
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  7. 2026A Universal Context-Reuse Layer for Cross-Model KV Sharing
  8. 2025Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding