mlsanityData to 7 Oct 2026

Understanding Stragglers in Large Model Training Using What-if Analysis

arXiv 2505.05713 ↗First posted 9 May 2025cs.DC cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) training is one of the most demanding distributed computations today, often requiring thousands of GPUs with frequent synchronization across machines. Such a workload pattern makes it susceptible to stragglers, where the training can be stalled by few slow workers. At ByteDance we find stragglers are not trivially always caused by hardware failures, but can arise from multiple complex factors. This work aims to present a comprehensive study on the straggler issues in LLM training, using a five-month trace collected from our ByteDance LLM training cluster. The core methodology is what-if analysis that simulates the scenario without any stragglers and contrasts with the actual case. We use this method to study the following questions: (1) how often do stragglers affect training jobs, and what effect do they have on job performance; (2) do stragglers exhibit temporal or spatial patterns; and (3) what are the potential root causes for stragglers?

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

17 of 39 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  7. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  8. 2017Revisiting Distributed Synchronous SGDcited by 142
  9. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  10. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  11. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
  12. 2024DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Modelscited by 16
5 more
  1. 2019Hop: Heterogeneity-Aware Decentralized Trainingcited by 10
  2. 2024Balancing Pipeline Parallelism with Vocabulary Parallelismcited by 6
  3. 2023DropCompute: simple and more robust distributed synchronous training via compute variance reductioncited by 4
  4. 2024FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Trainingcited by 4
  5. 2024Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  2. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  3. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  4. 2025Efficient AllReduce with Stragglerscited by 1
  5. 2025DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Servingcited by 1
  6. 2025Efficient Long-context Language Model Training by Core Attention Disaggregationcited by 0
  7. 2026From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUscited by 0
  8. 2026A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLMcited by 0
  9. 2026High-speed Networking for Giga-Scale AI Factoriescited by 0
  10. 2026SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Trainingcited by 0
  11. 2026The Life of a Token: from Words to Bits on the Wirecited by 0
  12. 2026A principled approach for energy-efficient training via phase-aware GPU frequency tuningcited by 0