mlsanityData to 7 Oct 2026

Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

arXiv 2603.05692 ↗First posted 5 Mar 2026cs.DC cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Breakthroughs in the generative AI domain have fueled an explosion of large language model (LLM)-powered applications, whose workloads fundamentally consist of sequences of inferences through transformer architectures. Within this rapidly expanding ecosystem, dense LLMs--those that activate all model parameters for each token generation--form the foundation for advanced expert-based variants. Dense models continue to dominate because of their strong generalization ability, scalability, ease of fine-tuning, and versatility across diverse tasks. In LLM inference systems, performance is mainly characterized by latency, response time, and throughput (i.e., tokens generated per unit of time). Latency and throughput are inherently coupled: optimizing for one often comes at the expense of the other. Moreover, batching strategies and parallelism configurations, which are essential when dense model parameters exceed device memory capacity, can significantly affect both latency and overall system throughput. This paper (i) investigates the workloads of two representative dense LLMs--Llama-3.1-70B and Llama-3.1-405B, focusing in particular on intra-node parallelization schemes, (ii) analyzes how input characteristics, batching, and parallelism strategies influence latency flexibility and the latency-throughput tradeoff, and (iii) identifies key performance bottlenecks that inform design choices for meeting service-level agreements (SLAs) and sustaining inference quality. Our empirical evaluations reveal that Tensor Parallelism (TP) improves the latency objectives while Pipeline Parallelism (PP) is better-suited for throughput-oriented applications. We highlight that their hybrid usage by controlling the TP and PP degrees provides control over the latency-throughput interplay.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 78 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

46 of 78 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
34 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  12. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  13. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  14. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  15. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  16. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  17. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  18. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  19. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  20. 2022Parallel Context Windows for Large Language Modelscited by 81
  21. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  22. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  23. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  24. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  25. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25
  26. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  27. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  28. 2025Context-Aware Hierarchical Merging for Long Document Summarizationcited by 9
  29. 2025Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batchingcited by 8
  30. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  31. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7
  32. 2024Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnectscited by 6
  33. 2024Understanding Data Movement in AMD Multi-GPU Systems with Infinity Fabriccited by 2
  34. 2024TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communicationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  3. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  4. 2023Splitwise: Efficient generative LLM inference using phase splitting
  5. 2023Efficient LLM Inference on CPUs
  6. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  7. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  8. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency