# Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

arXiv: https://arxiv.org/abs/2610.19405 · HTML: https://mlsanity.com/paper/2610.19405/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Builds on (most cited first, 30 of 34)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache](https://mlsanity.com/paper/2401.02669/) (2024)
- [Jenga: Effective Memory Management for Serving LLM with Heterogeneity](https://mlsanity.com/paper/2503.18292/) (2025)
- [Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving](https://mlsanity.com/paper/2508.01989/) (2025)
- [SLOs-Serve: Optimized Serving of Multi-SLO LLMs](https://mlsanity.com/paper/2504.08784/) (2025)
- [No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha](https://mlsanity.com/paper/2409.17264/) (2024)
- [Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving](https://mlsanity.com/paper/2507.06608/) (2025)
- [Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems](https://mlsanity.com/paper/2407.07000/) (2024)
- [Niyama : Breaking the Silos of LLM Inference Serving](https://mlsanity.com/paper/2503.22562/) (2025)
- [DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism](https://mlsanity.com/paper/2510.10620/) (2025)
- [DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving](https://mlsanity.com/paper/2504.09285/) (2025)
- [SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving](https://mlsanity.com/paper/2509.24626/) (2025)
- [FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving](https://mlsanity.com/paper/2602.22593/) (2026)
- [gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling](https://mlsanity.com/paper/2504.14775/) (2025)
- [Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads](https://mlsanity.com/paper/2509.16495/) (2025)
- [PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference](https://mlsanity.com/paper/2409.15104/) (2024)
- [Characterizing Communication Patterns in Distributed Large Language Model Inference](https://mlsanity.com/paper/2507.14392/) (2025)
- [Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism](https://mlsanity.com/paper/2605.05467/) (2026)
- [Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty](https://mlsanity.com/paper/2607.16892/) (2026)
- [Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide](https://mlsanity.com/paper/2602.09109/) (2026)
- [ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving](https://mlsanity.com/paper/2606.18741/) (2026)
- [VPP: Virtual Pipeline Parallelism for Efficient Chunked Prefill in Long-Context LLM Inference](https://mlsanity.com/paper/2608.26523/) (2026)
- [Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving](https://mlsanity.com/paper/2609.07883/) (2026)
- [Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking](https://mlsanity.com/paper/2602.21196/) (2026)
- [Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference](https://mlsanity.com/paper/2605.02329/) (2026)
- [Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP](https://mlsanity.com/paper/2605.08524/) (2026)
- [GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources](https://mlsanity.com/paper/2605.16867/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.