mlsanityData to 7 Oct 2026

Accelerating Distributed ML Training via Selective Synchronization

arXiv 2307.07950 ↗First posted 16 Jul 2023cs.DC cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

In distributed training, deep neural networks (DNNs) are launched over multiple workers concurrently and aggregate their local updates on each step in bulk-synchronous parallel (BSP) training. However, BSP does not linearly scale-out due to high communication cost of aggregation. To mitigate this overhead, alternatives like Federated Averaging (FedAvg) and Stale-Synchronous Parallel (SSP) either reduce synchronization frequency or eliminate it altogether, usually at the cost of lower final accuracy. In this paper, we present textttSelSync, a practical, low-overhead method for DNN training that dynamically chooses to incur or avoid communication at each step either by calling the aggregation op or applying local updates based on their significance. We propose various optimizations as part of textttSelSync to improve convergence in the context of textitsemi-synchronous training. Our system converges to the same or better accuracy than BSP while reducing training time by up to 14×.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 16 Jul 2023.

Builds on

37 of 53 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2018Federated Learning with Non-IID Datacited by 1,099
  11. 2016An overview of gradient descent optimization algorithmscited by 1,021
  12. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
25 more
  1. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  2. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  3. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  4. 2018An Empirical Model of Large-Batch Trainingcited by 268
  5. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  6. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  7. 2018The Convergence of Sparsified Gradient Methodscited by 228
  8. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  9. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  10. 2020The Early Phase of Neural Network Trainingcited by 114
  11. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  12. 2014Deep learning with Elastic Averaging SGDcited by 90
  13. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  14. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  15. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  16. 2019A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networkscited by 36
  17. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  18. 2020Accordion: Adaptive Gradient Communication via Critical Learning Regime Identificationcited by 19
  19. 2018Large batch size training of neural networks with adversarial training and second-order informationcited by 18
  20. 2020AdaScale SGD: A User-Friendly Algorithm for Distributed Trainingcited by 16
  21. 2022Adaptive Compression for Communication-Efficient Distributed Trainingcited by 14
  22. 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13
  23. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  24. 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
  25. 2023ScaDLES: Scalable Deep Learning over Streaming data at the Edgecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 5; 2026: 1.

7 papers.

  1. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  2. 2023Flexible Communication for Optimal Distributed Learning over Unpredictable Networkscited by 3
  3. 2025Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflationcited by 1
  4. 2025Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updatescited by 1
  5. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  6. 2025On Using Large-Batches in Federated Learningcited by 0
  7. 2026Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Trainingcited by 0