mlsanityData to 7 Oct 2026

On the Burstiness of Distributed Machine Learning Traffic

arXiv 2401.00329 ↗First posted 30 Dec 2023cs.LG cs.NIcited by 0

Not on the map. Why?

Abstract (arXiv)

Traffic from distributed training of machine learning (ML) models makes up a large and growing fraction of the traffic mix in enterprise data centers. While work on distributed ML abounds, the network traffic generated by distributed ML has received little attention. Using measurements on a testbed network, we investigate the traffic characteristics generated by the training of the ResNet-50 neural network with an emphasis on studying its short-term burstiness. For the latter we propose metrics that quantify traffic burstiness at different time scales. Our analysis reveals that distributed ML traffic exhibits a very high degree of burstiness on short time scales, exceeding a 60:1 peak-to-mean ratio on time intervals as long as 5~ms. We observe that training software orchestrates transmissions in such a way that burst transmissions from different sources within the same application do not result in congestion and packet losses. An extrapolation of the measurement data to multiple applications underscores the challenges of distributed ML traffic for congestion and flow control algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 66 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

17 of 66 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  5. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  6. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  7. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  8. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  9. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 37
  10. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  11. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  12. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21
5 more
  1. 2019Characterizing Deep Learning Training Workloads on Alibaba-PAIcited by 11
  2. 2020Characterizing and Modeling Distributed Training with Transient Cloud GPU Serverscited by 8
  3. 2022Impact of RoCE Congestion Control Policies on Distributed Training of DNNscited by 6
  4. 2018Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluationcited by 3
  5. 2022Analysis of Distributed Deep Learning in the Cloudcited by 1

Built on by

Not cited yet.