mlsanityData to 14 Oct 2026

Survey on Large Scale Neural Network Training

arXiv 2202.10435 ↗First posted 21 Feb 2022cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU device or can be trained using only a small per-GPU batch size. This survey provides a systematic overview of the approaches that enable more efficient DNNs training. We analyze techniques that save memory and make good use of computation and communication resources on architectures with a single or several GPUs. We summarize the main categories of strategies and compare strategies within and across categories. Along with approaches proposed in the literature, we discuss available implementations.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 59 references that are in the map. First posted on arXiv 21 Feb 2022.

Builds on

31 of 59 references.

References · 31

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2017Large Batch Training of Convolutional Networkscited by 535
  6. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  7. 2018Sparsified SGD with Memorycited by 376
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  9. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  10. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  11. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  12. 2018The Convergence of Sparsified Gradient Methodscited by 229
19 more
  1. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  2. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  3. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  4. 2016Memory-Efficient Backpropagation Through Timecited by 78
  5. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  6. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  7. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  8. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  9. 2020Dynamic Tensor Rematerializationcited by 46
  10. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  11. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  12. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  13. 2019Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memorycited by 22
  14. 2022Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reductioncited by 14
  15. 2018TFLMS: Large Model Support in TensorFlow by Graph Rewritingcited by 12
  16. 2019A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagationcited by 11
  17. 2017Divide-and-Conquer Checkpointing for Arbitrary Programs with No User Annotationcited by 6
  18. 2019Efficient Memory Management for GPU-based Deep Learning Systemscited by 6
  19. 2022Memory-Efficient Backpropagation through Large Linear Layerscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1.

4 papers.

  1. 2022Eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AIcited by 58
  2. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  3. 2024Quantization of Large Language Models with an Overdetermined Basiscited by 1
  4. 2025Slice or the Whole Pie? Utility Control for AI Modelscited by 0