mlsanityData to 14 Oct 2026

A Survey From Distributed Machine Learning to Distributed Deep Learning

arXiv 2307.05232 ↗First posted 11 Jul 2023cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence has made remarkable progress in handling complex tasks, thanks to advances in hardware acceleration and machine learning algorithms. However, to acquire more accurate outcomes and solve more complex issues, algorithms should be trained with more data. Processing this huge amount of data could be time-consuming and require a great deal of computation. To address these issues, distributed machine learning has been proposed, which involves distributing the data and algorithm across several machines. There has been considerable effort put into developing distributed machine learning algorithms, and different methods have been proposed so far. We divide these algorithms in classification and clustering (traditional machine learning), deep learning and deep reinforcement learning groups. Distributed deep learning has gained more attention in recent years and most of the studies have focused on this approach. Therefore, we mostly concentrate on this category. Based on the investigation of the mentioned algorithms, we highlighted the limitations that should be addressed in future research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 112 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

46 of 112 references.

References · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  7. 2017Deep Reinforcement Learning: An Overviewcited by 333
  8. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  9. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  10. 2018Distributed Prioritized Experience Replaycited by 269
  11. 2019A Survey on Distributed Machine Learningcited by 188
  12. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
34 more
  1. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  2. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  3. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  4. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 120
  5. 2014Deep learning with Elastic Averaging SGDcited by 90
  6. 2019Natural Compression for Distributed Deep Learningcited by 90
  7. 2017Device Placement Optimization with Reinforcement Learningcited by 86
  8. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  9. 2021Distributed Deep Learning in Open Collaborationscited by 58
  10. 2021Deep Reinforcement Learning in Computer Vision: A Comprehensive Surveycited by 43
  11. 2019Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Toolscited by 40
  12. 2018Decoupled Parallel Backpropagation with Convergence Guaranteecited by 39
  13. 2019Automated Machine Learning: State-of-The-Art and Open Challengescited by 29
  14. 2016Clustering Algorithms: A Comparative Approachcited by 28
  15. 2018Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platformcited by 28
  16. 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
  17. 2020Distributed Training of Deep Learning Models: A Taxonomic Perspectivecited by 25
  18. 2018Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networkscited by 19
  19. 2015Strategies and Principles of Distributed Machine Learning on Big Datacited by 16
  20. 2019HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Arraycited by 15
  21. 2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
  22. 2019Fully Decoupled Neural Network Learning Using Delayed Gradientscited by 12
  23. 2020The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelismcited by 11
  24. 2019Performance Analysis and Comparison of Distributed Machine Learning Systemscited by 9
  25. 2020Distributed Deep Reinforcement Learning: An Overviewcited by 9
  26. 2013Multi-GPU Training of ConvNetscited by 6
  27. 2019Distributed Deep Learning with Event-Triggered Communicationcited by 5
  28. 2021A Hybrid Parallelization Approach for Distributed and Scalable Deep Learningcited by 4
  29. 2022Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolboxcited by 4
  30. 2013Distributed k-means algorithmcited by 3
  31. 2018Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tilingcited by 3
  32. 2019Hybrid Data-Model Parallel Training for Sequence-to-Sequence Recurrent Neural Network Machine Translationcited by 3
  33. 2020DBS: Dynamic Batch Size For Distributed Deep Neural Network Trainingcited by 3
  34. 2022A Resilient Distributed Boosting Algorithmcited by 1

Built on by

Not cited yet.