mlsanityData to 14 Oct 2026

Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

arXiv 1711.04325 ↗First posted 12 Nov 2017cs.DC cs.CV cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 8; 2023: 6; 2024: 3; 2025: 0; 2026: 1.

22 papers.

Sort:

By area:

  1. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  2. 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
  3. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  4. 2022Auto-scaling Vision Transformers without Trainingcited by 15
  5. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  6. 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
  7. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  8. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11
  9. 2021Combining a Convolutional Neural Network with Autoencoders to Predict the Survival Chance of COVID-19 Patientscited by 5
  10. 2021PAFNet: An Efficient Anchor-Free Object Detector Guidancecited by 5
  11. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  12. 2024Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelismcited by 4
10 more
  1. 2022A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizescited by 3
  2. 2022Transfer learning and Local interpretable model agnostic based visual approach in Monkeypox Disease Detection and Classification: A Deep Learning insightscited by 3
  3. 2024I/O in Machine Learning Applications on HPC Systems: A 360-degree Surveycited by 3
  4. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  5. 2022Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGDcited by 1
  6. 2022Locally Asynchronous Stochastic Gradient Descent for Decentralised Deep Learningcited by 1
  7. 2023KAPLA: Pragmatic Representation and Fast Solving of Scalable NN Accelerator Dataflowcited by 0
  8. 2023Defect Analysis of 3D Printed Cylinder Object Using Transfer Learning Approachescited by 0
  9. 2023Towards Few-Annotation Learning in Computer Vision: Application to Image Classification and Object Detection taskscited by 0
  10. 2026SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networkscited by 0