# Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools

arXiv: https://arxiv.org/abs/1903.11314 · HTML: https://mlsanity.com/paper/1903.11314/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 30 of 40)

- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts](https://mlsanity.com/paper/2404.05019/) (2024)
- [Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learning](https://mlsanity.com/paper/2406.13846/) (2024)
- [The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey](https://mlsanity.com/paper/2305.13854/) (2023)
- [BAGUA: Scaling up Distributed Learning with System Relaxations](https://mlsanity.com/paper/2107.01499/) (2021)
- [An Invitation to Distributed Quantum Neural Networks](https://mlsanity.com/paper/2211.07056/) (2022)
- [Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching](https://mlsanity.com/paper/2305.12213/) (2023)
- [Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines](https://mlsanity.com/paper/2202.08679/) (2022)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [Brain-inspired Artificial Intelligence: A Comprehensive Review](https://mlsanity.com/paper/2408.14811/) (2024)
- [A Hybrid Architecture for Federated and Centralized Learning](https://mlsanity.com/paper/2105.03288/) (2021)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Optimising Communication Overhead in Federated Learning Using NSGA-II](https://mlsanity.com/paper/2204.02183/) (2022)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [Joint Optimization of Energy Consumption and Completion Time in Federated Learning](https://mlsanity.com/paper/2209.14900/) (2022)
- [Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training](https://mlsanity.com/paper/2303.06659/) (2023)
- [PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction](https://mlsanity.com/paper/2312.00839/) (2023)
- [MERCURY: Accelerating DNN Training By Exploiting Input Similarity](https://mlsanity.com/paper/2110.14904/) (2021)
- [Taking ROCKET on an Efficiency Mission: Multivariate Time Series Classification with LightWaveS](https://mlsanity.com/paper/2204.01379/) (2022)
- [Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges](https://mlsanity.com/paper/2211.14732/) (2022)
- [Deep Learning-based Techniques for Integrated Sensing and Communication Systems: State-of-the-Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06968/) (2025)
- [LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling](https://mlsanity.com/paper/2108.06629/) (2021)
- [AI Technical Considerations: Data Storage, Cloud usage and AI Pipeline](https://mlsanity.com/paper/2201.08356/) (2022)
- [TensAIR: Real-Time Training of Neural Networks from Data-streams](https://mlsanity.com/paper/2211.10280/) (2022)
- [Convergence Analysis of Decentralized ASGD](https://mlsanity.com/paper/2309.03754/) (2023)
- [A Comprehensive Survey on Deep Neural Image Deblurring](https://mlsanity.com/paper/2310.04719/) (2023)
- [Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey](https://mlsanity.com/paper/2406.08115/) (2024)
- [Going Forward-Forward in Distributed Deep Learning](https://mlsanity.com/paper/2404.08573/) (2024)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [Empowering Locally Deployable Medical Agent via State Enhanced Logical Skills for FHIR-based Clinical Tasks](https://mlsanity.com/paper/2603.06902/) (2026)

## Abstract (arXiv)

Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.