mlsanityData to 7 Oct 2026

A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks

arXiv 2111.04949 ↗First posted 9 Nov 2021cs.LG cs.AI cs.DCcited by 7

Not on the map. Why?

Abstract (arXiv)

The field of deep learning has witnessed a remarkable shift towards extremely compute- and memory-intensive neural networks. These newer larger models have enabled researchers to advance state-of-the-art tools across a variety of fields. This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators. In this paper, we discuss and compare current state-of-the-art frameworks for large scale distributed deep learning. First, we survey current practices in distributed learning and identify the different types of parallelism used. Then, we present empirical results comparing their performance on large image and language training tasks. Additionally, we address their statistical efficiency and memory consumption behavior. Based on our results, we discuss algorithmic and implementation portions of each framework which hinder performance.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 33 of its 71 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

33 of 71 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
21 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  2. 2018Object Detection with Deep Learning: A Reviewcited by 524
  3. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  4. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  5. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  7. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  8. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  9. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  10. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  11. 2017Revisiting Distributed Synchronous SGDcited by 142
  12. 2019MLPerf Training Benchmarkcited by 114
  13. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  14. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  15. 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
  16. 2019Message Passing Attention Networks for Document Understandingcited by 23
  17. 2019HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Arraycited by 15
  18. 2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
  19. 2020The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelismcited by 11
  20. 2019MIOpen: An Open Source Library For Deep Learning Primitivescited by 7
  21. 2020Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMAcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 1; 2026: 2.

7 papers.

  1. 2023OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workloadcited by 12
  2. 2023HPC-Coder: Modeling Parallel Programs using Large Language Modelscited by 11
  3. 2023Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methodscited by 2
  4. 2022Computational Complexity Evaluation of Neural Network Applications in Signal Processingcited by 1
  5. 2025GPU-centric Communication Schemes for HPC and ML Applicationscited by 0
  6. 2026A Non-Monotone Preconditioned Trust-Region Method for Neural Network Trainingcited by 0
  7. 2026The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Modelcited by 0