mlsanityData to 7 Oct 2026

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

arXiv 2007.01045 ↗First posted 2 Jul 2020cs.DCcited by 100

Not on the map. Why?

Abstract (arXiv)

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23x under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6x speedup of training throughput and reduces the memory consumption of 12% at the same time.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 10; 2023: 23; 2024: 22; 2025: 24; 2026: 10.

100 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  4. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  5. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  6. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  7. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  8. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  9. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  10. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  11. 2025Kling-Omni Technical Reportcited by 60
  12. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
38 more
  1. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  2. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  3. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  4. 2023Zero Bubble Pipeline Parallelismcited by 39
  5. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  6. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  7. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  8. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  9. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  10. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  11. 2024FuXi Weather: A data-to-forecast machine learning system for global weathercited by 21
  12. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  13. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  14. 2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
  15. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devicescited by 15
  16. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  17. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  18. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 14
  19. 2022AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awarenesscited by 13
  20. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  21. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  22. 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13
  23. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  24. 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
  25. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  26. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  27. 2024Pipeline Parallelism with Controllable Memorycited by 11
  28. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  29. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9
  30. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  31. 2024Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloadingcited by 7
  32. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  33. 2024PETRA: Parallel End-to-end Training with Reversible Architecturescited by 6
  34. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  35. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 6
  36. 2021End-to-end Adaptive Distributed Training on PaddlePaddlecited by 5
  37. 2024PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Trainingcited by 5
  38. 2025PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimizationcited by 5

50 most cited of 100 All 100 (JSON)