mlsanityData to 14 Oct 2026

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

arXiv 2110.03888 ↗First posted 8 Oct 2021cs.LG cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters. However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory footprint suffers from frustratingly low efficiency in model convergence. In this paper, we propose a simple training strategy called "Pseudo-to-Real" for high-memory-footprint-required large models. Pseudo-to-Real is compatible with large models with architecture of sequential layers. We demonstrate a practice of pretraining unprecedented 10-trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the application of Pseudo-to-Real, we also provide a technique, Granular CPU offloading, to manage CPU memory for training large model and maintain high GPU utilities. Fast training of extreme-scale models on a decent amount of resources can bring much smaller carbon footprint and contribute to greener AI.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

40 of 52 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
28 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2020On Layer Normalization in the Transformer Architecturecited by 922
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  13. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  14. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  15. 2018Universal Transformerscited by 647
  16. 2019Deep Equilibrium Modelscited by 615
  17. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  18. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  19. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  20. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  21. 2021Hash Layers For Large Sparse Modelscited by 237
  22. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  23. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  24. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  25. 2021M6: A Chinese Multimodal Pretrainercited by 91
  26. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  27. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  28. 2021M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformerscited by 36

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 14; 2023: 5; 2024: 7; 2025: 3.

31 papers.

Sort:

By area:

  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  2. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2023Challenges and Applications of Large Language Modelscited by 254
  6. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  8. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  9. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 109
  10. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  11. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  12. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
19 more
  1. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  2. 2024Large Language Models for Education: A Surveycited by 34
  3. 2023DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasetscited by 23
  4. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  5. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
  6. 2024DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Modelscited by 16
  7. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
  8. 2022Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inferencecited by 14
  9. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  10. 2024Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloadingcited by 7
  11. 2023BranchNorm: Robustly Scaling Extremely Deep Transformerscited by 6
  12. 2022Survey on Large Scale Neural Network Trainingcited by 4
  13. 2024M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretrainingcited by 4
  14. 2022Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithmscited by 2
  15. 2022COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Trainingcited by 2
  16. 2021AI in Human-computer Gaming: Techniques, Challenges and Opportunitiescited by 1
  17. 2025Efficient OpAmp Adaptation for Zoom Attention to Golden Contextscited by 1
  18. 2025Advances in Large Language Models for Medicinecited by 1
  19. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0