mlsanityData to 7 Oct 2026

Systems for Parallel and Distributed Large-Model Deep Learning Training

arXiv 2301.02691 ↗First posted 6 Jan 2023cs.DC cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep learning (DL) has transformed applications in a variety of domains, including computer vision, natural language processing, and tabular data analysis. The search for improved DL model accuracy has led practitioners to explore increasingly large neural architectures, with some recent Transformer models spanning hundreds of billions of learnable parameters. These designs have introduced new scale-driven systems challenges for the DL space, such as memory bottlenecks, poor runtime efficiency, and high costs of model development. Efforts to address these issues have explored techniques such as parallelization of neural architectures, spilling data across the memory hierarchy, and memory-efficient data representations. This survey will explore the large-model training systems landscape, highlighting key challenges and the various techniques that have been used to address them.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 6 Jan 2023.

Builds on

33 of 59 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  11. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  12. 2018Model compression via distillation and quantizationcited by 287
21 more
  1. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  2. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  3. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  4. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  5. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  6. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  7. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  8. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  9. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  10. 2019Low-Memory Neural Network Training: A Technical Reportcited by 56
  11. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  12. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
  13. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  14. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  15. 2020torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Modelscited by 26
  16. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  17. 2018TFLMS: Large Model Support in TensorFlow by Graph Rewritingcited by 12
  18. 2020LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentationcited by 8
  19. 2021Model-Parallel Model Selection for Deep Learning Systemscited by 6
  20. 2022Survey on Large Scale Neural Network Trainingcited by 4
  21. 2021Hydra: A System for Large Multi-Model Deep Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 0; 2025: 1.

4 papers.

  1. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  2. 2023InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Modelscited by 3
  3. 2023Saturn: An Optimized Data System for Large Model Deep Learning Workloadscited by 2
  4. 2025A Parallel Alternative for Energy-Efficient Neural Network Training and Inferencingcited by 0