# On the Burstiness of Distributed Machine Learning Traffic

arXiv: https://arxiv.org/abs/2401.00329 · HTML: https://mlsanity.com/paper/2401.00329/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 30 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale](https://mlsanity.com/paper/2303.14006/) (2023)
- [Machine Learning Model Sizes and the Parameter Gap](https://mlsanity.com/paper/2207.02852/) (2022)
- [Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems](https://mlsanity.com/paper/2003.09518/) (2020)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [TicTac: Accelerating Distributed Deep Learning with Communication Scheduling](https://mlsanity.com/paper/1803.03288/) (2018)
- [Priority-based Parameter Propagation for Distributed DNN Training](https://mlsanity.com/paper/1905.03960/) (2019)
- [Characterizing Deep Learning Training Workloads on Alibaba-PAI](https://mlsanity.com/paper/1910.05930/) (2019)
- [Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers](https://mlsanity.com/paper/2004.03072/) (2020)
- [Impact of RoCE Congestion Control Policies on Distributed Training of DNNs](https://mlsanity.com/paper/2207.10898/) (2022)
- [Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation](https://mlsanity.com/paper/1810.11112/) (2018)
- [Analysis of Distributed Deep Learning in the Cloud](https://mlsanity.com/paper/2208.14344/) (2022)

## Abstract (arXiv)

Traffic from distributed training of machine learning (ML) models makes up a large and growing fraction of the traffic mix in enterprise data centers. While work on distributed ML abounds, the network traffic generated by distributed ML has received little attention. Using measurements on a testbed network, we investigate the traffic characteristics generated by the training of the ResNet-50 neural network with an emphasis on studying its short-term burstiness. For the latter we propose metrics that quantify traffic burstiness at different time scales. Our analysis reveals that distributed ML traffic exhibits a very high degree of burstiness on short time scales, exceeding a 60:1 peak-to-mean ratio on time intervals as long as 5~ms. We observe that training software orchestrates transmissions in such a way that burst transmissions from different sources within the same application do not result in congestion and packet losses. An extrapolation of the measurement data to multiple applications underscores the challenges of distributed ML traffic for congestion and flow control algorithms.