mlsanityData to 7 Oct 2026

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

arXiv 2201.11990 ↗First posted 28 Jan 2022cs.CLcited by 557

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained general-purpose language models can achieve state-of-the-art accuracies in various natural language processing domains by adapting to downstream tasks via zero-shot, few-shot and fine-tuning techniques. Because of their success, the size of these models has increased rapidly, requiring high-performance hardware, software, and algorithmic techniques to enable training such large models. As the result of a joint effort between Microsoft and NVIDIA, we present details on the training of the largest monolithic transformer based language model, Megatron-Turing NLG 530B (MT-NLG), with 530 billion parameters. In this paper, we first focus on the infrastructure as well as the 3D parallelism methodology used to train this model using DeepSpeed and Megatron. Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model. Finally, we discuss various evaluation results, as well as other interesting observations and new properties exhibited by MT-NLG. We demonstrate that MT-NLG achieves superior zero-, one-, and few-shot learning accuracies on several NLP benchmarks and establishes new state-of-the-art results. We believe that our contributions will help further the development of large-scale training infrastructures, large-scale language models, and natural language generations.

Key facts. Cited by 557 papers in the map (data through 7 Oct 2026). Builds on 56 of its 78 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

56 of 78 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  13. 2017Mixed Precision Trainingcited by 1,135
  14. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  15. 2021Ethical and social risks of harm from Language Modelscited by 957
  16. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  17. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  18. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  19. 2017Dense-Captioning Events in Videoscited by 789
  20. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  21. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  22. 2019Defending Against Neural Fake Newscited by 553
  23. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  24. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  25. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  26. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  27. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  28. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  29. 2018A Simple Method for Commonsense Reasoningcited by 223
  30. 2016Exploring the Limits of Language Modelingcited by 172
  31. 2021Challenges in Detoxifying Language Modelscited by 169
  32. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  33. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  34. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  35. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  36. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  37. 2021M6: A Chinese Multimodal Pretrainercited by 91
  38. 2019Learning and Evaluating General Linguistic Intelligencecited by 78

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 132; 2023: 227; 2024: 125; 2025: 52; 2026: 21.

557 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
38 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  13. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  14. 2022Efficiently Scaling Transformer Inferencecited by 529
  15. 2023The Falcon Series of Open Language Modelscited by 526
  16. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  17. 2024Large Language Models: A Surveycited by 506
  18. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  19. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  20. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  21. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  22. 2023Scaling Data-Constrained Language Modelscited by 391
  23. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  24. 2023Language Models can Solve Computer Taskscited by 371
  25. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  26. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  27. 2022Teaching language models to support answers with verified quotescited by 290
  28. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  29. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  30. 2023Challenges and Applications of Large Language Modelscited by 251
  31. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  32. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  33. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  34. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  35. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  36. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  37. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  38. 2022Optimizing Prompts for Text-to-Image Generationcited by 204

50 most cited of 557 All 557 (JSON)