mlsanityData to 7 Oct 2026

MLCC: A Congestion Control Technique to Accelerate ML Training

arXiv 2402.09589 ↗First posted 14 Feb 2024cs.NI cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present MLCC, a novel technique to augment today's congestion control algorithms to accelerate DNN training jobs in shared GPU clusters in a fully distributed manner. At the heart of MLCC lies a straightforward principle: DNN training flows should scale their sending rate to shift other flows' communication into their compute periods, achieving interleaving. We show that integrating this principle into today's congestion control protocols is simple (requiring less than 60 lines of code for a given protocol) and enables DNN jobs to interleave within a few training iterations, thereby reducing network contention and improving job completion times. Our testbed demonstrates that MLCC accelerates the average and 99th percentile training iteration times by up to 1.9x and 2.7x respectively. Through extensive packet-level simulations, we observe a 1.35x improvement in training throughput on a 36-node, 288 GPU fat-tree topology.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 97 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

23 of 97 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Wide Residual Networkscited by 3,057
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  7. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  9. 2019CamemBERT: a Tasty French Language Modelcited by 368
  10. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  11. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  12. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
11 more
  1. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  2. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  3. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  4. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  5. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  6. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  7. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21
  8. 2022Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learningcited by 11
  9. 2018Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities (Complete Version)cited by 9
  10. 2020Scheduling Coflows with Dependency Graphcited by 1
  11. 2023Isolated Scheduling for Distributed Training Tasks in GPU Clusterscited by 1

Built on by

Not cited yet.