mlsanityData to 14 Oct 2026

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

arXiv 2402.15627 ↗First posted 23 Feb 2024cs.LG cs.DCcited by 112

Not on the map. Why?

Abstract (arXiv)

We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Builds on 36 of its 69 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

36 of 69 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2020Longformer: The Long-Document Transformercited by 3,413
24 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  13. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  14. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  15. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  16. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  17. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  18. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  19. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  20. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  21. 2018Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Trainingcited by 28
  22. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  23. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21
  24. 2022WeLM: A Well-Read Pre-trained Language Model for Chinesecited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 37; 2025: 49; 2026: 26.

112 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  4. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 71
  5. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  6. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61
  7. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  8. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  9. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  10. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  11. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  12. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
38 more
  1. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  2. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  3. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  4. 2024PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inferencecited by 28
  5. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 28
  6. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  7. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 24
  8. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
  9. 2025Robust LLM Training Infrastructure at ByteDancecited by 16
  10. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  11. 2024Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitationcited by 14
  12. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  13. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 14
  14. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  15. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  16. 2024Cloud Atlas: Efficient Fault Localization for Cloud Systems using Language Models and Causal Insightcited by 12
  17. 2024ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Trainingcited by 12
  18. 2025Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environmentscited by 12
  19. 2024A Survey on Failure Analysis and Fault Injection in AI Systemscited by 10
  20. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUscited by 10
  21. 2024Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Paralleliscited by 9
  22. 2024Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Studycited by 9
  23. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 9
  24. 2024Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learningcited by 8
  25. 2024Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledgecited by 8
  26. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  27. 2024LocalValueBench: A Collaboratively Built and Extensible Benchmark for Evaluating Localized Value Alignment and Ethical Safety in Large Language Modelscited by 7
  28. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  29. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
  30. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7
  31. 2025Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputerscited by 7
  32. 2025Optimizing Question Semantic Space for Dynamic Retrieval-Augmented Multi-hop Question Answeringcited by 6
  33. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  34. 2026Training LLMs with Fault Tolerant HSDP on 100,000 GPUscited by 6
  35. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  36. 2025BlueLM-2.5-3B Technical Reportcited by 5
  37. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  38. 2024Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimizationcited by 4

50 most cited of 112 All 112 (JSON)