mlsanityData to 14 Oct 2026

Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

arXiv 2206.04959 ↗First posted 10 Jun 2022cs.LG cs.DC cs.MAcited by 11

Not on the map. Why?

Abstract (arXiv)

Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the training process is extremely memory-intensive and communication-intensive. These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency. To achieve this goal, some custom software frameworks such as Megatron-LM and DeepSpeed are developed. However, current 3D parallelism frameworks still meet two issues: i) they are not transparent to model developers, which need to manually modify the model to parallelize training. ii) their utilization of computation, GPU memory and network bandwidth are not sufficient. We propose Merak, an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys with an automatic model partitioner, which uses a graph sharding algorithm on a proxy representation of the model. Merak also presents the non-intrusive API for scaling out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine in Merak. It uses several techniques to exploit available training resources, including shifted critical path pipeline schedule that brings a higher computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs show Merak can speedup the training performance over the state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42X, 1.39X, 1.43X, and 1.61X, respectively.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 10 Jun 2022.

Builds on

33 of 55 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
21 more
  1. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  2. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  3. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  4. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  5. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  6. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  7. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  8. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  9. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  10. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  11. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  12. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  13. 2020Dynamic Tensor Rematerializationcited by 46
  14. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  15. 2021Torch.fx: Practical Program Capture and Transformation for Deep Learning in Pythoncited by 40
  16. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  17. 2021OneFlow: Redesign the Distributed Deep Learning Framework from Scratchcited by 15
  18. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  19. 2020Pipelined Backpropagation at Scale: Training Large Models without Batchescited by 11
  20. 2021Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Trainingcited by 11
  21. 2021End-to-end Adaptive Distributed Training on PaddlePaddlecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 4; 2026: 2.

11 papers.

  1. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  2. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  3. 2023A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUscited by 6
  4. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  5. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 3
  6. 2024Optimizing Large Model Training through Overlapped Activation Recomputationcited by 2
  7. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  8. 2024Real-time and Downtime-tolerant Fault Diagnosis for Railway Turnout Machines (RTMs) Empowered with Cloud-Edge Pipeline Parallelismcited by 0
  9. 2025Exploiting Block Coordinate Descent for Cost-Effective LLM Model Trainingcited by 0
  10. 2026Piper: A Programmable Distributed Training Systemcited by 0
  11. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Trainingcited by 0