mlsanityData to 7 Oct 2026

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

arXiv 1706.02677 ↗First posted 8 Jun 2017cs.CV cs.DC cs.LGcited by 1,479

Not on the map. Why?

Cited by 39 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ~90% scaling efficiency when moving from 8 to 256 GPUs. Our findings enable training visual recognition models on internet-scale data with high efficiency.

Key facts. Cited by 1,479 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 344; 2022: 320; 2023: 271; 2024: 220; 2025: 181; 2026: 141.

1,479 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,158
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2022Visual Prompt Tuningcited by 1,674
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2021Multiscale Vision Transformerscited by 896
38 more
  1. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  4. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  5. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  8. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  9. 2025Mean Flows for One-step Generative Modelingcited by 524
  10. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  11. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  12. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  13. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  14. 2021Early Convolutions Help Transformers See Bettercited by 395
  15. 2023Llemma: An Open Language Model For Mathematicscited by 385
  16. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  17. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  18. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  19. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  20. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  21. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  22. 2022How Do Vision Transformers Work?cited by 305
  23. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  24. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  25. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  26. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  27. 2022Vision Transformers for Single Image Dehazingcited by 255
  28. 2022Autoformalization with Large Language Modelscited by 252
  29. 2023A Cookbook of Self-Supervised Learningcited by 248
  30. 2022Masked Visual Pre-training for Motor Controlcited by 246
  31. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  32. 2021WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognitioncited by 221
  33. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  34. 2023Improving CLIP Training with Language Rewritescited by 216
  35. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  36. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  37. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  38. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202

50 most cited of 1,479 All 1,479 (JSON)