mlsanityData to 7 Oct 2026

PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers

arXiv 2605.02189 ↗First posted 4 May 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Offline LLM inference seeks to maximize request processing under fixed budgets, making commodity GPU servers a promising choice. However, prior work typically considers offloading and parallelism in isolation, resulting in suboptimal performance. In this paper, we propose PipeMax, a high-throughput LLM inference system that integrates pipeline parallelism with offloading to overcome interconnect and memory constraints on GPU servers. Particularly, pipeline parallelism naturally incurs low communication overhead and keeps only one batch active on each GPU at a time, which enables offloading the KV cache of inactive batches. By coordinating computation with offloading data movement, PipeMax effectively expands GPU memory capacity and sustains large-batch execution. Experiments show that PipeMax achieves up to 2.51x higher throughput than vLLM, and up to 1.42x and 1.38x higher throughput than state-of-the-art high-throughput LLM systems, respectively, on an 8-GPU node.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 24 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

15 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding9 Mar
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 11

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  6. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  7. 2024Optimizing LLM Queries in Relational Data Analytics Workloadscited by 20
  8. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  9. 2025TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inferencecited by 8
  10. 2025PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimizationcited by 5
  11. 2025EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestrationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  2. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  3. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  4. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  5. 2025PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
  6. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  7. 2026DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference ↗
  8. 2025LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices