mlsanityData to 7 Oct 2026

Large-Scale Deep Learning Optimizations: A Comprehensive Survey

arXiv 2111.00856 ↗First posted 1 Nov 2021cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Deep learning have achieved promising results on a wide spectrum of AI applications. Larger datasets and models consistently yield better performance. However, we generally spend longer training time on more computation and communication. In this survey, we aim to provide a clear sketch about the optimizations for large-scale deep learning with regard to the model accuracy and model efficiency. We investigate algorithms that are most commonly used for optimizing, elaborate the debatable topic of generalization gap arises in large-batch training, and review the SOTA strategies in addressing the communication overhead and reducing the memory footprints.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 87 of its 125 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

87 of 125 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2017Mask R-CNNcited by 7,259
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2015Fast R-CNNcited by 2,930
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  12. 2017Neural Collaborative Filteringcited by 1,108
  13. 2016An overview of gradient descent optimization algorithmscited by 1,021
  14. 2020On Layer Normalization in the Transformer Architecturecited by 906
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  16. 2019On the Convergence of Adam and Beyondcited by 834
  17. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  18. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  19. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  20. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  21. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  22. 2017Large Batch Training of Convolutional Networkscited by 529
  23. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  24. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  25. 2014New insights and perspectives on the natural gradient methodcited by 435
  26. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  27. 2018Sparsified SGD with Memorycited by 373
  28. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  29. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  30. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  31. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  32. 2017Three Factors Influencing Minima in SGDcited by 270
  33. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  34. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  35. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  36. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  37. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  38. 2018The Convergence of Sparsified Gradient Methodscited by 228

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 1; 2025: 1; 2026: 1.

10 papers.

  1. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  2. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 34
  3. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  4. 2022CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPUcited by 4
  5. 2022Refiner: Data Refining against Gradient Leakage Attacks in Federated Learningcited by 2
  6. 2025Synthetic Human Action Video Data Generation with Pose Transfercited by 2
  7. 2022Benchmark Assessment for DeepSpeed Optimization Librarycited by 1
  8. 2024Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimizationcited by 1
  9. 2023Accurate and Fast Fischer-Tropsch Reaction Microkinetics using PINNscited by 0
  10. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0