mlsanityData to 14 Oct 2026

COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training

arXiv 2211.16648 ↗First posted 30 Nov 2022cs.DC cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Modern Deep Learning (DL) models have grown to sizes requiring massive clusters of specialized, high-end nodes to train. Designing such clusters to maximize both performance and utilization--to amortize their steep cost--is a challenging task requiring careful balance of compute, memory, and network resources. Moreover, a plethora of each model's tuning knobs drastically affect the performance, with optimal values often depending on the underlying cluster's characteristics, which necessitates a complex cluster-workload co-design process. To facilitate the design space exploration of such massive DL training clusters, we introduce COMET, a holistic cluster design methodology and workflow to jointly study the impact of parallelization strategies and key cluster resource provisioning on the performance of distributed DL training. We develop a step-by-step process to establish a reusable and flexible methodology, and demonstrate its application with case studies of training large models on cluster configurations of variable compute, memory, and network resources. Our case studies demonstrate COMET's utility in identifying promising architectural optimization directions and guiding system designers in configuring key model and cluster parameters. To illustrate, cluster configuration comparisons identify performance differences of up to 7.7x and highlight performance optimization opportunities of up to 1.4x when employing memory expansion as an optimization technique.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 86 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

33 of 86 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  10. 2018Deep Interest Evolution Network for Click-Through Rate Predictioncited by 338
  11. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  12. 2018An Empirical Model of Large-Batch Trainingcited by 281
21 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  2. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  3. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  4. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  5. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  6. 2019The Architectural Implications of Facebook's DNN-based Personalized Recommendationcited by 83
  7. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  8. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  9. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  10. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 38
  11. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  12. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  13. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 16
  14. 2019Characterizing Deep Learning Training Workloads on Alibaba-PAIcited by 11
  15. 2020Hierarchical Roofline Performance Analysis for Deep Learning Applicationscited by 11
  16. 2022TPP: Transparent Page Placement for CXL-Enabled Tiered-Memorycited by 10
  17. 2020Time-Based Roofline for Deep Learning Performance Analysiscited by 7
  18. 2022Impact of RoCE Congestion Control Policies on Distributed Training of DNNscited by 6
  19. 2019Performance Analysis of Deep Learning Workloads on Leading-edge Systemscited by 4
  20. 20208 Steps to 3.7 TFLOP/s on NVIDIA V100 GPU: Roofline Analysis and Other Trickscited by 2
  21. 2022Ridgeline: A 2D Roofline Model for Distributed Systemscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  2. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14