# Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour

arXiv: https://arxiv.org/abs/2011.00071 · HTML: https://mlsanity.com/paper/2011.00071/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2020.

## Built on by (most cited first, 3 of 3)

- [Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters](https://mlsanity.com/paper/2108.03645/) (2021)
- [A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculture](https://mlsanity.com/paper/2210.11318/) (2022)
- [DARTSRepair: Core-failure-set Guided DARTS for Network Robustness to Common Corruptions](https://mlsanity.com/paper/2209.10381/) (2022)

## Abstract (arXiv)

EfficientNets are a family of state-of-the-art image classification models based on efficiently scaled convolutional neural networks. Currently, EfficientNets can take on the order of days to train; for example, training an EfficientNet-B0 model takes 23 hours on a Cloud TPU v2-8 node. In this paper, we explore techniques to scale up the training of EfficientNets on TPU-v3 Pods with 2048 cores, motivated by speedups that can be achieved when training at such scales. We discuss optimizations required to scale training to a batch size of 65536 on 1024 TPU-v3 cores, such as selecting large batch optimizers and learning rate schedules as well as utilizing distributed evaluation and batch normalization techniques. Additionally, we present timing and performance benchmarks for EfficientNet models trained on the ImageNet dataset in order to analyze the behavior of EfficientNets at scale. With our optimizations, we are able to train EfficientNet on ImageNet to an accuracy of 83% in 1 hour and 4 minutes.