mlsanityData to 14 Oct 2026

TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

arXiv 2102.07988 ↗First posted 16 Feb 2021cs.LG cs.CL cs.DCcited by 75

Not on the map. Why?

Abstract (arXiv)

Model parallelism has become a necessity for training modern large-scale deep language models. In this work, we identify a new and orthogonal dimension from existing model parallel approaches: it is possible to perform pipeline parallelism within a single training sequence for Transformer-based language models thanks to its autoregressive property. This enables a more fine-grained pipeline compared with previous work. With this key idea, we design TeraPipe, a high-performance token-level pipeline parallel algorithm for synchronous model-parallel training of Transformer-based language models. We develop a novel dynamic programming-based algorithm to calculate the optimal pipelining execution scheme given a specific model and cluster configuration. We show that TeraPipe can speed up the training by 5.0x for the largest GPT-3 model with 175 billion parameters on an AWS cluster with 48 p3.16xlarge instances compared with state-of-the-art model-parallel methods. The code for reproduction can be found at https://github.com/zhuohan123/terapipe

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

22 of 39 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020Reformer: The Efficient Transformercited by 1,668
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
10 more
  1. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  2. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  3. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  4. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  5. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  6. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  7. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  8. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  9. 2018Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networkscited by 23
  10. 2016Optimizing Performance of Recurrent Neural Networks on GPUscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 9; 2023: 14; 2024: 15; 2025: 19; 2026: 7.

75 papers.

Sort:

By area:

  1. 2021Ethical and social risks of harm from Language Modelscited by 966
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  7. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  8. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  9. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  10. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  11. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  12. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
38 more
  1. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  2. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  3. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  4. 2024PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inferencecited by 28
  5. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  6. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  7. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  8. 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
  9. 2022On Optimizing the Communication of Model Parallelismcited by 19
  10. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  11. 2025WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Trainingcited by 17
  12. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  13. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devicescited by 15
  14. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  15. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  16. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
  17. 2022AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awarenesscited by 13
  18. 2023Does compressing activations help model parallel training?cited by 12
  19. 2025Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learningcited by 12
  20. 2021Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Trainingcited by 11
  21. 2022AI and 6G into the Metaverse: Fundamentals, Challenges and Future Research Trendscited by 8
  22. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 8
  23. 2024SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compilecited by 7
  24. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  25. 2025SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Trainingcited by 6
  26. 2021End-to-end Adaptive Distributed Training on PaddlePaddlecited by 5
  27. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
  28. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  29. 2025PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimizationcited by 5
  30. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5
  31. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  32. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  33. 2022Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledgecited by 3
  34. 2023TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networkscited by 3
  35. 2023InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Modelscited by 3
  36. 2025LoRAFusion: Efficient LoRA Fine-Tuning for LLMscited by 3
  37. 2023Baechi: Fast Device Placement of Machine Learning Graphscited by 2
  38. 2023Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platformcited by 2

50 most cited of 75 All 75 (JSON)