mlsanityData to 7 Oct 2026

torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

arXiv 2004.09910 ↗First posted 21 Apr 2020cs.DC cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

We design and implement a ready-to-use library in PyTorch for performing micro-batch pipeline parallelism with checkpointing proposed by GPipe (Huang et al., 2019). In particular, we develop a set of design components to enable pipeline-parallel gradient computation in PyTorch's define-by-run and eager execution environment. We show that each component is necessary to fully benefit from pipeline parallelism in such environment, and demonstrate the efficiency of the library by applying it to various network architectures including AmoebaNet-D and U-Net. Our library is available at https://github.com/kakaobrain/torchgpipe .

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 1; 2023: 7; 2024: 8; 2025: 4.

26 papers.

Sort:

  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  2. 2023A Survey on Efficient Training of Transformerscited by 37
  3. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  4. 2024ORBIT: Oak Ridge Base Foundation Model for Earth System Predictabilitycited by 17
  5. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  6. 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13
  7. 2025ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscalingcited by 10
  8. 2024Linear Attention Sequence Parallelismcited by 8
  9. 2021A Survey and Empirical Evaluation of Parallel Deep Learning Frameworkscited by 7
  10. 2021Automatic Graph Partitioning for Very Large-scale Deep Learningcited by 6
  11. 2024PETRA: Parallel End-to-end Training with Reversible Architecturescited by 6
  12. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
14 more
  1. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 6
  2. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  3. 2021Hydra: A System for Large Multi-Model Deep Learningcited by 3
  4. 2021An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networkscited by 2
  5. 2023Saturn: An Optimized Data System for Large Model Deep Learning Workloadscited by 2
  6. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 2
  7. 2022PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networkscited by 1
  8. 2023RTP: Rethinking Tensor Parallelism with Memory Deduplicationcited by 1
  9. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  10. 2023A Survey From Distributed Machine Learning to Distributed Deep Learningcited by 0
  11. 2023Practical Performance Guarantees for Pipelined DNN Inferencecited by 0
  12. 2024FreeRide: Harvesting Bubbles in Pipeline Parallelismcited by 0
  13. 2024On the Convergence Theory of Pipeline Gradient-based Analog In-memory Trainingcited by 0
  14. 2025Distributed Deep Learning using Stochastic Gradient Stalenesscited by 0