mlsanityData to 14 Oct 2026

Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective

arXiv 2509.10371 ↗First posted 12 Sep 2025cs.DC cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The rapid scaling of Large Language Models (LLMs) has pushed training workloads far beyond the limits of single-node analysis, demanding a deeper understanding of how these models behave across large-scale, multi-GPU systems. In this paper, we present a comprehensive characterization of LLM training across diverse real-world workloads and hardware platforms, including NVIDIA H100/H200 and AMD MI250 GPUs. We analyze dense and sparse models under various parallelism strategies -- tensor, pipeline, data, and expert -- and evaluate their effects on hardware utilization, power consumption, and thermal behavior. We further evaluate the effectiveness of optimizations such as activation recomputation and compute-communication overlap. Our findings show that performance is not determined solely by scaling hardware capacity. Scale-up systems with fewer, higher-memory GPUs can outperform scale-out systems in communication-bound regimes, but only under carefully tuned configurations; in other cases, scale-out deployments achieve superior throughput. We also show that certain parallelism combinations, such as tensor with pipeline, lead to bandwidth underutilization due to inefficient data chunking, while increasing microbatch sizes beyond a certain point induces bursty execution and peak power excursions that worsen thermal throttling. These insights reveal how training performance is shaped by complex interactions between hardware, system topology, and model execution. We conclude by offering recommendations for system and hardware design to improve the scalability and reliability of future LLM systems and workloads. The source code of this project is available at https://github.com/sitar-lab/CharLLM-PPT.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 40 of its 84 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

40 of 84 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2022Efficiently Scaling Transformer Inferencecited by 539
28 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  3. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  4. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  5. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  6. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  7. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  8. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  9. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  10. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  11. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  12. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  13. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  14. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  15. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  16. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  17. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
  18. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  19. 2021Accelerating Recommendation System Training by Leveraging Popular Choicescited by 22
  20. 2023FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropoutcited by 16
  21. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  22. 2023Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 13
  23. 2024SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compilecited by 7
  24. 2025Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputerscited by 7
  25. 2015Dissecting GPU Memory Hierarchy through Microbenchmarkingcited by 6
  26. 2024Integrated Hardware Architecture and Device Placement Searchcited by 4
  27. 2025Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implicationscited by 3
  28. 2024Workload-Aware Hardware Accelerator Mining for Distributed Deep Learning Trainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025From FLOPs to Footprints: The Resource Cost of Artificial Intelligencecited by 2
  2. 2026ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Servingcited by 1
  3. 2026NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learningcited by 0
  4. 2026MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 0
  5. 2026The Life of a Token: from Words to Bits on the Wirecited by 0