mlsanityData to 14 Oct 2026

Efficient AllReduce with Stragglers

arXiv 2505.23523 ↗First posted 29 May 2025cs.LG cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

Distributed machine learning workloads use data and tensor parallelism for training and inference, both of which rely on the AllReduce collective to synchronize gradients or activations. However, AllReduce algorithms are delayed by the slowest GPU to reach the synchronization barrier before the collective (i.e., the straggler). To address this challenge, we propose StragglAR: a parallel algorithm for AllReduce that accelerates distributed training and inference by exploiting natural variation in GPU execution times. StragglAR implements a ReduceScatter among the remaining GPUs during the straggler-induced delay, and then executes a novel collective algorithm to complete the AllReduce once the final GPU reaches the synchronization barrier. StragglAR achieves a 2x theoretical speedup over popular bandwidth-efficient algorithms for large GPU clusters, surpassing the lower bound for bandwidth-optimal synchronous AllReduce by leveraging the asymmetry in when GPUs reach the synchronization barrier. On an 8-GPU server, StragglAR provides a 25% speedup over state-of-the-art AllReduce algorithms.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 45 of its 97 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

45 of 97 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  12. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
33 more
  1. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  2. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  3. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  4. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  5. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  6. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  7. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  8. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  9. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  10. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  11. 2020Synthesizing Optimal Collective Algorithmscited by 30
  12. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  13. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  14. 2017Straggler Mitigation in Distributed Optimization Through Data Encodingcited by 26
  15. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  16. 2023Epidemic Learning: Boosting Decentralized Learning with Randomized Communicationcited by 21
  17. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  18. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  19. 2023FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropoutcited by 16
  20. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 16
  21. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  22. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operationscited by 15
  23. 2024Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneitycited by 13
  24. 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
  25. 2024SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validationcited by 11
  26. 2023OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloudcited by 8
  27. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  28. 2024On the Optimal Time Complexities in Decentralized Stochastic Asynchronous Optimizationcited by 8
  29. 2023A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learningcited by 7
  30. 2022GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Modelcited by 4
  31. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  32. 2023DropCompute: simple and more robust distributed synchronous training via compute variance reductioncited by 4
  33. 2024FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Don't Let a Few Network Failures Slow the Entire AllReducecited by 0