mlsanityData to 7 Oct 2026

Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision

arXiv 2205.11913 ↗First posted 24 May 2022cs.DC cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Deep learning (DL) shows its prosperity in a wide variety of fields. The development of a DL model is a time-consuming and resource-intensive procedure. Hence, dedicated GPU accelerators have been collectively constructed into a GPU datacenter. An efficient scheduler design for such GPU datacenter is crucially important to reduce the operational cost and improve resource utilization. However, traditional approaches designed for big data or high performance computing workloads can not support DL workloads to fully utilize the GPU resources. Recently, substantial schedulers are proposed to tailor for DL workloads in GPU datacenters. This paper surveys existing research efforts for both training and inference workloads. We primarily present how existing schedulers facilitate the respective workloads from the scheduling objectives and resource consumption features. Finally, we prospect several promising future research directions. More detailed summary with the surveyed paper and code links can be found at our project website: https://github.com/S-Lab-System-Group/Awesome-DL-Scheduling-Papers

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 59 of its 179 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

59 of 179 references.

References · 59

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  4. 2017Deep Learning for Medical Image Analysiscited by 399
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  6. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  7. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  8. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  9. 2019A Survey on Distributed Machine Learningcited by 187
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  11. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  12. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
38 more
  1. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  2. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  3. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  4. 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
  5. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  6. 2019Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Toolscited by 40
  7. 2018Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implicationscited by 35
  8. 2018Rafiki: Machine Learning as an Analytics Service Systemcited by 33
  9. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  10. 2018PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systemscited by 22
  11. 2020Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendationscited by 21
  12. 2020IOS: Inter-Operator Scheduler for CNN Accelerationcited by 20
  13. 2017Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloadscited by 16
  14. 2019DL2: A Deep Learning-driven Scheduler for Deep Learning Clusterscited by 14
  15. 2021Solving Large-Scale Granular Resource Allocation Problems Efficiently with POPcited by 13
  16. 2022Dynamic GPU Energy Optimization for Machine Learning Training Workloadscited by 12
  17. 2018Dynamic Space-Time Scheduling for GPU Inferencecited by 11
  18. 2019Characterizing Deep Learning Training Workloads on Alibaba-PAIcited by 11
  19. 2020Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analyticscited by 11
  20. 2019Privacy Accounting and Quality Control in the Sage Differentially Private ML Platformcited by 10
  21. 2021JIZHI: A Fast and Cost-Effective Model-As-A-Service System for Web-Scale Online Inference at Baiducited by 10
  22. 2017Serving deep learning models in a serverless platformcited by 9
  23. 2018Online Job Scheduling in Distributed Machine Learning Clusterscited by 9
  24. 2019One Size Does Not Fit All: Quantifying and Exposing the Accuracy-Latency Trade-off in Machine Learning Cloud Service APIs via Tolerance Tierscited by 7
  25. 2021Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusterscited by 7
  26. 2021Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problemcited by 7
  27. 2022A Survey of Multi-Tenant Deep Learning Inference on GPUcited by 7
  28. 2019BARISTA: Efficient and Scalable Serverless Serving System for Deep Learning Prediction Servicescited by 6
  29. 2020HyperSched: Dynamic Resource Reallocation for Model Development on a Deadlinecited by 6
  30. 2019Perseus: Characterizing Performance and Cost of Multi-Tenant Serving for CNN Modelscited by 5
  31. 2020Hysia: Serving DNN-Based Video-to-Retail Applications in Cloudcited by 5
  32. 2021A Sum-of-Ratios Multi-Dimensional-Knapsack Decomposition for DNN Resource Schedulingcited by 5
  33. 2021A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunitiescited by 5
  34. 2018TrIMS: Transparent and Isolated Model Sharing for Low Latency Deep LearningInference in Function as a Service Environmentscited by 4
  35. 2019FfDL : A Flexible Multi-tenant Deep Learning Platformcited by 4
  36. 2020Effective Elastic Scaling of Deep Learning Workloadscited by 4
  37. 2021Privacy Budget Schedulingcited by 4
  38. 2022GADGET: Online Resource Optimization for Scheduling Ring-All-Reduce Learning Jobscited by 4

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 4.

6 papers.

  1. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  2. 2025Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environmentscited by 4
  3. 2023Sustainable AIGC Workload Scheduling of Geo-Distributed Data Centers: A Multi-Agent Reinforcement Learning Approachcited by 3
  4. 2025Accurate GPU Memory Prediction for Deep Learning Jobs through Dynamic Analysiscited by 0
  5. 2025AEGIS: Runtime-Guided GPU Collocation for Multi-Tenant Deep Learning Trainingcited by 0
  6. 2025Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusterscited by 0