mlsanityData to 7 Oct 2026

Beyond Data and Model Parallelism for Deep Neural Networks

arXiv 1807.05358 ↗First posted 14 Jul 2018cs.DCcited by 177

Not on the map. Why?

Abstract (arXiv)

The computational requirements for training deep neural networks (DNNs) have grown to the point that it is now standard practice to parallelize training. Existing deep learning systems commonly use data or model parallelism, but unfortunately, these strategies often result in suboptimal parallelization performance. In this paper, we define a more comprehensive search space of parallelization strategies for DNNs called SOAP, which includes strategies to parallelize a DNN in the Sample, Operation, Attribute, and Parameter dimensions. We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine. To accelerate this search, FlexFlow introduces a novel execution simulator that can accurately predict a parallelization strategy's performance and is three orders of magnitude faster than prior approaches that have to execute each strategy. We evaluate FlexFlow with six real-world DNN benchmarks on two GPU clusters and show that FlexFlow can increase training throughput by up to 3.8x over state-of-the-art approaches, even when including its search time, and also improves scalability.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 36; 2022: 27; 2023: 43; 2024: 34; 2025: 19; 2026: 17.

177 papers.

Sort:

By area:

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  6. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  7. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  8. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  9. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  10. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  12. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
38 more
  1. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  2. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  3. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  4. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  5. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  6. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  7. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  8. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  9. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  10. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  11. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  12. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  13. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  14. 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 46
  15. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  16. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  17. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 39
  18. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  19. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  20. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  21. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  22. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  23. 2021A Runtime-Based Computational Performance Predictor for Deep Neural Network Trainingcited by 29
  24. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  25. 2022Flashlight: Enabling Innovation in Tools for Machine Learningcited by 25
  26. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  27. 2022Efficient Neural Network Analysis with Sum-of-Infeasibilitiescited by 24
  28. 2021Accelerating Recommendation System Training by Leveraging Popular Choicescited by 22
  29. 2021CoSA: Scheduling by Constrained Optimization for Spatial Acceleratorscited by 22
  30. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  31. 2024AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoisingcited by 22
  32. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  33. 2022On Optimizing the Communication of Model Parallelismcited by 19
  34. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  35. 2021Distributed Learning Systems with First-order Methodscited by 17
  36. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  37. 2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
  38. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 15

50 most cited of 177 All 177 (JSON)