mlsanityData to 7 Oct 2026

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

arXiv 1909.08053 ↗First posted 17 Sep 2019cs.CLcited by 1,829

Not on the map. Why?

Cited by 198 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).

Key facts. Cited by 1,829 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 140; 2022: 159; 2023: 251; 2024: 358; 2025: 461; 2026: 458.

1,829 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2023A Survey of Large Language Modelscited by 2,537
38 more
  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  13. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  16. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  17. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  18. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  19. 2025Qwen-Image Technical Reportcited by 973
  20. 2023BloombergGPT: A Large Language Model for Financecited by 923
  21. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  22. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  23. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  24. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  25. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  26. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  27. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  28. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  29. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  30. 2024Genie: Generative Interactive Environmentscited by 706
  31. 2024Yi: Open Foundation Models by 01.AIcited by 706
  32. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  33. 2023A Comprehensive Overview of Large Language Modelscited by 639
  34. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  35. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  36. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  37. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  38. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595

50 most cited of 1,829 All 1,829 (JSON)