mlsanityData to 7 Oct 2026

Benchmark Assessment for DeepSpeed Optimization Library

arXiv 2202.12831 ↗First posted 12 Feb 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Deep Learning (DL) models are widely used in machine learning due to their performance and ability to deal with large datasets while producing high accuracy and performance metrics. The size of such datasets and the complexity of DL models cause such models to be complex, consuming large amount of resources and time to train. Many recent libraries and applications are introduced to deal with DL complexity and efficiency issues. In this paper, we evaluated one example, Microsoft DeepSpeed library through classification tasks. DeepSpeed public sources reported classification performance metrics on the LeNet architecture. We extended this through evaluating the library on several modern neural network architectures, including convolutional neural networks (CNNs) and Vision Transformer (ViT). Results indicated that DeepSpeed, while can make improvements in some of those cases, it has no or negative impact on others.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 12 Feb 2022.

Builds on

23 of 38 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  8. 2017A Closer Look at Memorization in Deep Networkscited by 936
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  12. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
11 more
  1. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  2. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  3. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  4. 2019Adaptive Attention Span in Transformerscited by 148
  5. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  6. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  7. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  8. 2018On exponential convergence of SGD in non-convex over-parametrized learningcited by 33
  9. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  10. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  11. 2018BigDL: A Distributed Deep Learning Framework for Big Datacited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Modelscited by 12