Data to 7 Oct 2026

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

arXiv 1802.09941 ↗First posted 26 Feb 2018cs.LG cs.CV cs.DCcited by 154

Not on the map. Why?

Abstract (arXiv)

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 45; 2022: 20; 2023: 34; 2024: 21; 2025: 15; 2026: 18.

154 papers.

Sort:

By area:

2021Pre-Trained Models: Past, Present and Futurecited by 319
2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 55
2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36

38 more

2021Vector Symbolic Architectures as a Computing Framework for Emerging Hardwarecited by 35
2021DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregationcited by 31
2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
2021Performance of a Geometric Deep Learning Pipeline for HL-LHC Particle Trackingcited by 29
2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
2020PFL-MoE: Personalized Federated Learning Based on Mixture of Expertscited by 21
2022Merlin HugeCTR: GPU-accelerated Recommender System Training and Inferencecited by 20
2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
2021Distributed Learning Systems with First-order Methodscited by 17
2021BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learningcited by 16
2021Truly Sparse Neural Networks at Scalecited by 15
2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
2021Clairvoyant Prefetching for Distributed Machine Learning I/Ocited by 14
2023Machine learning and domain decomposition methods -- a surveycited by 14
2022An Invitation to Distributed Quantum Neural Networkscited by 13
2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
2023The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Surveycited by 13
2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
2023HPC-Coder: Modeling Parallel Programs using Large Language Modelscited by 11
2023Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimizationcited by 11
2021GIST: Distributed Training for Large-Scale Graph Convolutional Networkscited by 10
2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
2022A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Modelscited by 9
2023DeepPCR: Parallelizing Sequential Operations in Neural Networkscited by 9
2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9
2021COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compressioncited by 8
2021AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learningcited by 8
2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
2022Anomaly Detection through Unsupervised Federated Learningcited by 8

50 most cited of 154 All 154 (JSON)