mlsanityData to 7 Oct 2026

Image Classification at Supercomputer Scale

arXiv 1811.06992 ↗First posted 16 Nov 2018cs.LG cs.DC stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Deep learning is extremely computationally intensive, and hardware vendors have responded by building faster accelerators in large clusters. Training deep learning models at petaFLOPS scale requires overcoming both algorithmic and systems software challenges. In this paper, we discuss three systems-related optimizations: (1) distributed batch normalization to control per-replica batch sizes, (2) input pipeline optimizations to sustain model throughput, and (3) 2-D torus all-reduce to speed up gradient summation. We combine these optimizations to train ResNet-50 on ImageNet to 76.3% accuracy in 2.2 minutes on a 1024-chip TPU v3 Pod with a training throughput of over 1.05 million images/second and no accuracy drop.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 6; 2023: 1; 2024: 0; 2025: 1; 2026: 1.

16 papers.

Sort:

  1. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  2. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  3. 2021Tesseract: Parallelize the Tensor Parallelism Efficientlycited by 19
  4. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11
  5. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  6. 2022Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelinescited by 10
  7. 2021Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniquescited by 9
  8. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  9. 2022Disjoint Masking with Joint Distillation for Efficient Masked Image Modelingcited by 6
  10. 2022CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Frameworkcited by 5
  11. 2022Large-batch Optimization for Dense Visual Predictionscited by 4
  12. 2021Representation range needs for 16-bit neural network trainingcited by 1
4 more
  1. 2023Spreeze: High-Throughput Parallel Reinforcement Learning Frameworkcited by 1
  2. 2026Prototype Transformer: Towards Language Model Architectures Interpretable by Designcited by 1
  3. 2022Leveraging the HW/SW Optimizations and Ecosystems that Drive the AI Revolutioncited by 0
  4. 2025ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanismcited by 0