# Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

arXiv: https://arxiv.org/abs/1711.04325 · HTML: https://mlsanity.com/paper/1711.04325/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2017.

## Built on by (most cited first, 22 of 22)

- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [FFCV: Accelerating Training by Removing Data Bottlenecks](https://mlsanity.com/paper/2306.12517/) (2023)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Auto-scaling Vision Transformers without Training](https://mlsanity.com/paper/2202.11921/) (2022)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)
- [Numerical Optimizations for Weighted Low-rank Estimation on Language Model](https://mlsanity.com/paper/2211.09718/) (2022)
- [Knowledge Distillation in Vision Transformers: A Critical Review](https://mlsanity.com/paper/2302.02108/) (2023)
- [Concurrent Adversarial Learning for Large-Batch Training](https://mlsanity.com/paper/2106.00221/) (2021)
- [Combining a Convolutional Neural Network with Autoencoders to Predict the Survival Chance of COVID-19 Patients](https://mlsanity.com/paper/2104.08954/) (2021)
- [PAFNet: An Efficient Anchor-Free Object Detector Guidance](https://mlsanity.com/paper/2104.13534/) (2021)
- [Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods](https://mlsanity.com/paper/2406.13936/) (2024)
- [Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism](https://mlsanity.com/paper/2412.21124/) (2024)
- [A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizes](https://mlsanity.com/paper/2210.12184/) (2022)
- [Transfer learning and Local interpretable model agnostic based visual approach in Monkeypox Disease Detection and Classification: A Deep Learning insights](https://mlsanity.com/paper/2211.05633/) (2022)
- [I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey](https://mlsanity.com/paper/2404.10386/) (2024)
- [Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Training](https://mlsanity.com/paper/2211.08544/) (2022)
- [Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGD](https://mlsanity.com/paper/2203.06638/) (2022)
- [Locally Asynchronous Stochastic Gradient Descent for Decentralised Deep Learning](https://mlsanity.com/paper/2203.13085/) (2022)
- [KAPLA: Pragmatic Representation and Fast Solving of Scalable NN Accelerator Dataflow](https://mlsanity.com/paper/2306.15676/) (2023)
- [Defect Analysis of 3D Printed Cylinder Object Using Transfer Learning Approaches](https://mlsanity.com/paper/2310.08645/) (2023)
- [Towards Few-Annotation Learning in Computer Vision: Application to Image Classification and Object Detection tasks](https://mlsanity.com/paper/2311.04888/) (2023)
- [SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks](https://mlsanity.com/paper/2606.02927/) (2026)

## Abstract (arXiv)

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.