# MLCC: A Congestion Control Technique to Accelerate ML Training

arXiv: https://arxiv.org/abs/2402.09589 · HTML: https://mlsanity.com/paper/2402.09589/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 97 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training](https://mlsanity.com/paper/2110.14883/) (2021)
- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning](https://mlsanity.com/paper/2008.12260/) (2020)
- [Themis: Fair and Efficient GPU Cluster Scheduling](https://mlsanity.com/paper/1907.01484/) (2019)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs](https://mlsanity.com/paper/2202.00433/) (2022)
- [TicTac: Accelerating Distributed Deep Learning with Communication Scheduling](https://mlsanity.com/paper/1803.03288/) (2018)
- [CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters](https://mlsanity.com/paper/2308.00852/) (2023)
- [Priority-based Parameter Propagation for Distributed DNN Training](https://mlsanity.com/paper/1905.03960/) (2019)
- [Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning](https://mlsanity.com/paper/2210.00093/) (2022)
- [Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities (Complete Version)](https://mlsanity.com/paper/1803.09615/) (2018)
- [Scheduling Coflows with Dependency Graph](https://mlsanity.com/paper/2012.11702/) (2020)
- [Isolated Scheduling for Distributed Training Tasks in GPU Clusters](https://mlsanity.com/paper/2308.05692/) (2023)

## Abstract (arXiv)

We present MLCC, a novel technique to augment today's congestion control algorithms to accelerate DNN training jobs in shared GPU clusters in a fully distributed manner. At the heart of MLCC lies a straightforward principle: DNN training flows should scale their sending rate to shift other flows' communication into their compute periods, achieving interleaving. We show that integrating this principle into today's congestion control protocols is simple (requiring less than 60 lines of code for a given protocol) and enables DNN jobs to interleave within a few training iterations, thereby reducing network contention and improving job completion times. Our testbed demonstrates that MLCC accelerates the average and 99th percentile training iteration times by up to 1.9x and 2.7x respectively. Through extensive packet-level simulations, we observe a 1.35x improvement in training throughput on a 36-node, 288 GPU fat-tree topology.