mlsanityData to 7 Oct 2026

PipeMare: Asynchronous Pipeline Parallel DNN Training

arXiv 1910.05124 ↗First posted 9 Oct 2019cs.DC cs.LG stat.MLcited by 54

Not on the map. Why?

Abstract (arXiv)

Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices are necessary. We devise PipeMare, a simple yet robust training method that tolerates asynchronous updates during PP execution without sacrificing utilization or memory, which allows efficient use of fine-grained pipeline parallelism. Concretely, when tested on ResNet and Transformer networks, asynchrony enables PipeMare to use up to 2.7× less memory or get 4.3× higher pipeline utilization, with similar model quality, when compared to state-of-the-art synchronous PP training techniques.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 7; 2022: 13; 2023: 15; 2024: 8; 2025: 4; 2026: 6.

54 papers.

Sort:

By area:

  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  3. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  4. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  5. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  6. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  7. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  8. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  9. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  10. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  11. 2023Zero Bubble Pipeline Parallelismcited by 39
  12. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
38 more
  1. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  2. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  3. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  4. 2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
  5. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  6. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 14
  7. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  8. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  9. 2021Automap: Towards Ergonomic Automated Parallelism for ML Modelscited by 12
  10. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  11. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  12. 2024Pipeline Parallelism with Controllable Memorycited by 11
  13. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  14. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  15. 2024PETRA: Parallel End-to-end Training with Reversible Architecturescited by 6
  16. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  17. 2025Nesterov Method for Asynchronous Pipeline Parallel Optimizationcited by 6
  18. 2023Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Trainingcited by 5
  19. 2021Parareal Neural Networks Emulating a Parallel-in-time Algorithmcited by 4
  20. 2022Survey on Large Scale Neural Network Trainingcited by 4
  21. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  22. 2023PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Predictioncited by 4
  23. 2021Hydra: A System for Large Multi-Model Deep Learningcited by 3
  24. 2022Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learningcited by 2
  25. 2022Automatic Discovery of Composite SPMD Partitioning Strategies in PartIRcited by 2
  26. 2023Baechi: Fast Device Placement of Machine Learning Graphscited by 2
  27. 2022PiPar: Pipeline Parallelism for Collaborative Machine Learningcited by 2
  28. 2023Saturn: An Optimized Data System for Large Model Deep Learning Workloadscited by 2
  29. 2024Ravnest: Decentralized Asynchronous Training on Heterogeneous Devicescited by 2
  30. 2026AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelismcited by 2
  31. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  32. 2023Exploring the Robustness of Decentralized Training for Large Language Modelscited by 1
  33. 2025Coded Deep Learning: Framework and Algorithmcited by 1
  34. 2023Practical Performance Guarantees for Pipelined DNN Inferencecited by 0
  35. 2024Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networkscited by 0
  36. 20242BP: 2-Stage Backpropagationcited by 0
  37. 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communicationscited by 0
  38. 2025Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Modelscited by 0

50 most cited of 54 All 54 (JSON)