Dive into Big Model Training
Not on the map. Why?
Abstract (arXiv)
The increasing scale of model size and continuous improvement of performance herald the arrival of the Big Model era. In this report, we explore what and how the big model training works by diving into training objectives and training methodologies. Specifically,training objectives describe how to leverage web-scale data to develop extremely capable and incredibly large models based on self-supervised learning, and training methodologies which are based on distributed training describe how to make big model training a reality. We summarize the existing training methodologies into three main categories: training parallelism, memory-saving technologies, and model sparsity design. Training parallelism can be categorized into data, pipeline, and tensor parallelism according to the dimension of parallelism that takes place. Memory-saving technologies are orthogonal and complementary to training parallelism. And model sparsity design further scales up the model size with a constant computational cost. A continuously updated paper list of big model training is provided at https://github.com/qhliu26/BM-Training.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 61 of its 92 references that are in the map. First posted on arXiv 25 Jul 2022.
Builds on
61 of 92 references.
References · 61
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
38 more
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2017Neural Discrete Representation Learningcited by 4,712
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
- 2017Mixed Precision Trainingcited by 1,156
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
- 2016Training Deep Nets with Sublinear Memory Costcited by 848
- 2016Pixel Recurrent Neural Networkscited by 816
- 2020Self-supervised Learning: Generative or Contrastivecited by 694
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
- 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
- 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
- 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
- 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
- 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
- 2017Revisiting Distributed Synchronous SGDcited by 142
- 2020Sparse GPU Kernels for Deep Learningcited by 141
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.