mlsanityData to 7 Oct 2026

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

arXiv 2104.04473 ↗First posted 9 Apr 2021cs.CL cs.DCcited by 621

Not on the map. Why?

Cited by 67 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have led to state-of-the-art accuracies across a range of tasks. However, training these models efficiently is challenging for two reasons: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required to train these models can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to fundamental scaling issues at thousands of GPUs, e.g., due to expensive cross-node communication or devices spending significant time waiting on other devices to make progress. In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters. We survey techniques for pipeline parallelism and propose a novel interleaved pipeline parallelism schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. We quantitatively study the trade-offs between tensor, pipeline, and data parallelism, and provide intuition as to how to configure distributed training of a large model. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs with achieved per-GPU throughput of 52% of theoretical peak. Our code is open sourced at https://github.com/nvidia/megatron-lm.

Key facts. Cited by 621 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 9 Apr 2021.

Builds on

27 of 39 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
15 more
  1. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  2. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  3. 2019MLPerf Training Benchmarkcited by 114
  4. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  5. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  6. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  7. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  8. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  9. 2013Petuum: A New Platform for Distributed Machine Learning on Big Datacited by 35
  10. 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
  11. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  12. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  13. 2017Integrated Model, Batch and Domain Parallelism in Training Neural Networkscited by 16
  14. 2019Scale MLPerf-0.6 models on Google TPU-v3 Podscited by 14
  15. 2020Pipelined Backpropagation at Scale: Training Large Models without Batchescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 43; 2023: 96; 2024: 163; 2025: 166; 2026: 147.

621 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
38 more
  1. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  2. 2025Kimi K2: Open Agentic Intelligencecited by 523
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  7. 2023Scaling Data-Constrained Language Modelscited by 391
  8. 2025Kimi-VL Technical Reportcited by 381
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  10. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  12. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  13. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  14. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  15. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  16. 2024Corrective Retrieval Augmented Generationcited by 187
  17. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  18. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  19. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  20. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  21. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  22. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  23. 2024Nemotron-4 340B Technical Reportcited by 143
  24. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  25. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  26. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  27. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  28. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  29. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  30. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  31. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  32. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  33. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  34. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  35. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  36. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  37. 2026Kimi K3: Open Frontier Intelligencecited by 99
  38. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98

50 most cited of 621 All 621 (JSON)