mlsanityData to 14 Oct 2026

Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools

arXiv 1903.11314 ↗First posted 27 Mar 2019cs.DC cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 12; 2023: 10; 2024: 8; 2025: 3; 2026: 2.

40 papers.

Sort:

By area:

  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  2. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  3. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  4. 2023The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Surveycited by 14
  5. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  6. 2022An Invitation to Distributed Quantum Neural Networkscited by 13
  7. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  8. 2022Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelinescited by 10
  9. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
  10. 2024Brain-inspired Artificial Intelligence: A Comprehensive Reviewcited by 9
  11. 2021A Hybrid Architecture for Federated and Centralized Learningcited by 8
  12. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
28 more
  1. 2022Optimising Communication Overhead in Federated Learning Using NSGA-IIcited by 7
  2. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  3. 2022Joint Optimization of Energy Consumption and Completion Time in Federated Learningcited by 5
  4. 2023Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Trainingcited by 5
  5. 2023PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Predictioncited by 4
  6. 2021MERCURY: Accelerating DNN Training By Exploiting Input Similaritycited by 3
  7. 2022Taking ROCKET on an Efficiency Mission: Multivariate Time Series Classification with LightWaveScited by 3
  8. 2022Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challengescited by 3
  9. 2025Deep Learning-based Techniques for Integrated Sensing and Communication Systems: State-of-the-Art, Challenges, and Opportunitiescited by 3
  10. 2021LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Schedulingcited by 2
  11. 2022AI Technical Considerations: Data Storage, Cloud usage and AI Pipelinecited by 2
  12. 2022TensAIR: Real-Time Training of Neural Networks from Data-streamscited by 2
  13. 2023Convergence Analysis of Decentralized ASGDcited by 2
  14. 2023A Comprehensive Survey on Deep Neural Image Deblurringcited by 2
  15. 2024Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Surveycited by 2
  16. 2024Going Forward-Forward in Distributed Deep Learningcited by 1
  17. 2025Onboard Optimization and Learning: A Surveycited by 1
  18. 2026Empowering Locally Deployable Medical Agent via State Enhanced Logical Skills for FHIR-based Clinical Taskscited by 1
  19. 2021Responsive parallelized architecture for deploying deep learning models in production environmentscited by 0
  20. 2022ACE: Towards Application-Centric Edge-Cloud Collaborative Intelligencecited by 0
  21. 2022Implicit Channel Learning for Machine Learning Applications in 6G Wireless Networkscited by 0
  22. 2022MEESO: A Multi-objective End-to-End Self-Optimized Approach for Automatically Building Deep Learning Modelscited by 0
  23. 2023Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networkscited by 0
  24. 2023A Survey From Distributed Machine Learning to Distributed Deep Learningcited by 0
  25. 2023The Landscape of Modern Machine Learning: A Review of Machine, Distributed and Federated Learningcited by 0
  26. 2024A resource-efficient model for deep kernel learningcited by 0
  27. 2025Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Reviewcited by 0
  28. 2026Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Trainingcited by 0