mlsanityData to 14 Oct 2026

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

arXiv 2610.19405 ↗First posted 14 Oct 2026cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 38 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

34 of 38 references.

References · 34

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  5. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  6. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  7. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  8. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  9. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  10. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 13
  11. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systemscited by 10
  12. 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 7
22 more
  1. 2025DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelismcited by 7
  2. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 6
  3. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 6
  4. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 6
  5. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 5
  6. 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloadscited by 5
  7. 2024PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inferencecited by 4
  8. 2025Characterizing Communication Patterns in Distributed Large Language Model Inferencecited by 3
  9. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelismcited by 3
  10. 2026Robust KV Cache Management for LLM Serving under Output Token Length Uncertaintycited by 3
  11. 2026Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guidecited by 2
  12. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Servingcited by 2
  13. 2026VPP: Virtual Pipeline Parallelism for Efficient Chunked Prefill in Long-Context LLM Inferencecited by 2
  14. 2026Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Servingcited by 2
  15. 2026Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunkingcited by 1
  16. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inferencecited by 1
  17. 2026Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCPcited by 1
  18. 2026GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resourcescited by 1
  19. 2026FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Trainingcited by 1
  20. 2026Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Acceleratorscited by 1
  21. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 1
  22. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offscited by 1

Built on by

Not cited yet.