mlsanityData to 14 Oct 2026

Maximizing Parallelism in Distributed Training for Huge Neural Networks

arXiv 2105.14450 ↗First posted 30 May 2021cs.DC cs.LG cs.PFcited by 23

Not on the map. Why?

Abstract (arXiv)

The recent Natural Language Processing techniques have been refreshing the state-of-the-art performance at an incredible speed. Training huge language models is therefore an imperative demand in both industry and academy. However, huge language models impose challenges to both hardware and software. Graphical processing units (GPUs) are iterated frequently to meet the exploding demand, and a variety of ASICs like TPUs are spawned. However, there is still a tension between the fast growth of the extremely huge models and the fact that Moore's law is approaching the end. To this end, many model parallelism techniques are proposed to distribute the model parameters to multiple devices, so as to alleviate the tension on both memory and computation. Our work is the first to introduce a 3-dimensional model parallelism for expediting huge language models. By reaching a perfect load balance, our approach presents smaller memory and communication cost than existing state-of-the-art 1-D and 2-D model parallelism. Our experiments on 64 TACC's V100 GPUs show that our 3-D parallelism outperforms the 1-D and 2-D parallelism with 2.32x and 1.57x speedup, respectively.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 30 May 2021.

Builds on

15 of 25 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  10. 2017Large Batch Training of Convolutional Networkscited by 535
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  12. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
3 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  2. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  3. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 8; 2024: 5; 2025: 5; 2026: 1.

23 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Efficient Large Language Models: A Surveycited by 186
  3. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  4. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  5. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  6. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  7. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  8. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  9. 2024LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMscited by 12
  10. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  11. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  12. 2023A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUscited by 6
11 more
  1. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  2. 2023Rethinking Memory and Communication Cost for Efficient Large Language Model Trainingcited by 4
  3. 2024Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Modelscited by 4
  4. 2025A Survey on Memory-Efficient Transformer-Based Model Training in AI for Sciencecited by 3
  5. 2025TrainVerify: Equivalence-Based Verification for Distributed LLM Trainingcited by 3
  6. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  7. 2022Dive into Big Model Trainingcited by 1
  8. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  9. 2024Mixed Sparsity Training: Achieving 4× FLOP Reduction for Transformer Pretrainingcited by 0
  10. 2024Comprehensive Performance Modeling and System Design Insights for Foundation Modelscited by 0
  11. 2025You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Modelscited by 0