mlsanityData to 7 Oct 2026

HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

arXiv 2005.14038 ↗First posted 28 May 2020cs.DCcited by 46

Not on the map. Why?

Abstract (arXiv)

Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 7; 2023: 6; 2024: 12; 2025: 9; 2026: 4.

46 papers.

Sort:

By area:

  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  3. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  4. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  5. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  6. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  7. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  8. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  9. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  10. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  11. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  12. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
34 more
  1. 2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
  2. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  3. 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13
  4. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  5. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  6. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  7. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  8. 2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
  9. 2021Automatic Graph Partitioning for Very Large-scale Deep Learningcited by 6
  10. 2022Demand Layering for Real-Time DNN Inference with Minimized Memory Usagecited by 6
  11. 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
  12. 2023Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chipscited by 5
  13. 2022Survey on Large Scale Neural Network Trainingcited by 4
  14. 2022Enabling All In-Edge Deep Learning: A Literature Reviewcited by 4
  15. 2025SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networkscited by 4
  16. 2024QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devicescited by 3
  17. 2022A Multiagent Framework for the Asynchronous and Collaborative Extension of Multitask ML Systemscited by 2
  18. 2023Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Searchcited by 2
  19. 2024Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Surveycited by 2
  20. 2024Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computingcited by 1
  21. 2024PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Timecited by 1
  22. 2025KAITIAN: A Unified Communication Framework for Enabling Efficient Collaboration Across Heterogeneous Accelerators in Embodied AI Systemscited by 1
  23. 2025CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networkscited by 1
  24. 2026HetCCL: Accelerating LLM Training with Heterogeneous GPUscited by 1
  25. 2023Toward efficient resource utilization at edge nodes in federated learningcited by 0
  26. 2023Practical Performance Guarantees for Pipelined DNN Inferencecited by 0
  27. 2025Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Trainingcited by 0
  28. 2025MoLink: Distributed and Efficient Serving Framework for Large Modelscited by 0
  29. 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communicationscited by 0
  30. 2025Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usagecited by 0
  31. 2025TeamFormer: Shallow Parallel Transformers with Progressive Approximationcited by 0
  32. 2026A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variabilitycited by 0
  33. 2026HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusterscited by 0
  34. 2026CogGuard: Cognitive and Operational Profiling for Proactive Warning in Edge Intelligent Servicescited by 0