mlsanityData to 7 Oct 2026

Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey

arXiv 2406.08115 ↗First posted 12 Jun 2024cs.DC cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

With rapidly increasing distributed deep learning workloads in large-scale data centers, efficient distributed deep learning framework strategies for resource allocation and workload scheduling have become the key to high-performance deep learning. The large-scale environment with large volumes of datasets, models, and computational and communication resources raises various unique challenges for resource allocation and workload scheduling in distributed deep learning, such as scheduling complexity, resource and workload heterogeneity, and fault tolerance. To uncover these challenges and corresponding solutions, this survey reviews the literature, mainly from 2019 to 2024, on efficient resource allocation and workload scheduling strategies for large-scale distributed DL. We explore these strategies by focusing on various resource types, scheduling granularity levels, and performance goals during distributed training and inference processes. We highlight critical challenges for each topic and discuss key insights of existing technologies. To illustrate practical large-scale resource allocation and workload scheduling in real distributed deep learning scenarios, we use a case study of training large language models. This survey aims to encourage computer science, artificial intelligence, and communications researchers to understand recent advances and explore future research directions for efficient framework strategies for large-scale distributed deep learning.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 141 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

45 of 141 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 225
33 more
  1. 2020Communication-Efficient Edge AI: Algorithms and Systemscited by 95
  2. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  3. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  4. 2017Deep Packet: A Novel Approach For Encrypted Traffic Classification Using Deep Learningcited by 69
  5. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  6. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  7. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  8. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  9. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  10. 2018A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Securitycited by 42
  11. 2019Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Toolscited by 40
  12. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  13. 2018Deep Learning with Long Short-Term Memory for Time Series Predictioncited by 39
  14. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  15. 2018Rafiki: Machine Learning as an Analytics Service Systemcited by 33
  16. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  17. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  18. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  19. 2020Communication-Efficient Distributed Deep Learning: A Comprehensive Surveycited by 25
  20. 2022Task-Oriented Communications for NextG: End-to-End Deep Learning and AI Security Aspectscited by 20
  21. 2023Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiencycited by 20
  22. 2019Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applicationscited by 17
  23. 2020Interlocking Backpropagation: Improving depthwise model-parallelismcited by 17
  24. 2022iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloudcited by 13
  25. 2022Dynamic GPU Energy Optimization for Machine Learning Training Workloadscited by 12
  26. 2020DNN-based Localization from Channel Estimates: Feature Design and Experimental Resultscited by 10
  27. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  28. 2023MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusterscited by 7
  29. 2023FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inferencecited by 6
  30. 2019MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learningcited by 5
  31. 2021A Sum-of-Ratios Multi-Dimensional-Knapsack Decomposition for DNN Resource Schedulingcited by 5
  32. 2019FfDL : A Flexible Multi-tenant Deep Learning Platformcited by 4
  33. 2023DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipeliningcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environmentscited by 4
  2. 2025Secure Resource Allocation via Constrained Deep Reinforcement Learningcited by 1