# ConvNets Match Vision Transformers at Scale

arXiv: https://arxiv.org/abs/2310.16764 · HTML: https://mlsanity.com/paper/2310.16764/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Augment your batch: better training with larger batches](https://mlsanity.com/paper/1901.09335/) (2019)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)

## Built on by (most cited first, 16 of 16)

- [Better than classical? The subtle art of benchmarking quantum machine learning models](https://mlsanity.com/paper/2403.07059/) (2024)
- [Transformers without Normalization](https://mlsanity.com/paper/2503.10622/) (2025)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Understanding and Minimising Outlier Features in Neural Network Training](https://mlsanity.com/paper/2405.19279/) (2024)
- [AstroPT: Scaling Large Observation Models for Astronomy](https://mlsanity.com/paper/2405.14930/) (2024)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [PMFSNet: Polarized Multi-scale Feature Self-attention Network For Lightweight Medical Image Segmentation](https://mlsanity.com/paper/2401.07579/) (2024)
- [EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping](https://mlsanity.com/paper/2409.07480/) (2024)
- [Progressive trajectory matching for medical dataset distillation](https://mlsanity.com/paper/2403.13469/) (2024)
- [Navigating Scaling Laws: Compute Optimality in Adaptive Model Training](https://mlsanity.com/paper/2311.03233/) (2023)
- [HyenaPixel: Global Image Context with Convolutions](https://mlsanity.com/paper/2402.19305/) (2024)
- [Examining the Impact of Optical Aberrations to Image Classification and Object Detection Models](https://mlsanity.com/paper/2504.18510/) (2025)
- [GeoPos: A Minimal Positional Encoding for Enhanced Fine-Grained Details in Image Synthesis Using Convolutional Neural Networks](https://mlsanity.com/paper/2401.01951/) (2024)
- [LaFAM: Unsupervised Feature Attribution with Label-free Activation Maps](https://mlsanity.com/paper/2407.06059/) (2024)
- [Structured Event Reasoning with Large Language Models](https://mlsanity.com/paper/2408.16098/) (2024)
- [CCNeXt: An Effective Self-Supervised Stereo Depth Estimation Approach](https://mlsanity.com/paper/2509.22627/) (2025)

## Abstract (arXiv)

Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale. We challenge this belief by evaluating a performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset of images often used for training foundation models. We consider pre-training compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a series of networks of increasing depth and width from the NFNet model family. We observe a log-log scaling law between held out loss and compute budget. After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers with comparable compute budgets. Our strongest fine-tuned model achieves a Top-1 accuracy of 90.4%.