mlsanityData to 7 Oct 2026

ConvNets Match Vision Transformers at Scale

arXiv 2310.16764 ↗First posted 25 Oct 2023cs.CV cs.LG cs.NEcited by 16

Not on the map. Why?

Abstract (arXiv)

Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale. We challenge this belief by evaluating a performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset of images often used for training foundation models. We consider pre-training compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a series of networks of increasing depth and width from the NFNet model family. We observe a log-log scaling law between held out loss and compute budget. After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers with comparable compute budgets. Our strongest fine-tuned model achieves a Top-1 accuracy of 90.4%.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 12 of its 15 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

12 of 15 references.

References · 12

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2021Scaling Vision Transformerscited by 993
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  11. 2019Augment your batch: better training with larger batchescited by 37
  12. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 3; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  2. 2025Transformers without Normalizationcited by 69
  3. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  4. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  5. 2024AstroPT: Scaling Large Observation Models for Astronomycited by 9
  6. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  7. 2024PMFSNet: Polarized Multi-scale Feature Self-attention Network For Lightweight Medical Image Segmentationcited by 6
  8. 2024EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotypingcited by 6
  9. 2024Progressive trajectory matching for medical dataset distillationcited by 5
  10. 2023Navigating Scaling Laws: Compute Optimality in Adaptive Model Trainingcited by 4
  11. 2024HyenaPixel: Global Image Context with Convolutionscited by 2
  12. 2025Examining the Impact of Optical Aberrations to Image Classification and Object Detection Modelscited by 2
4 more
  1. 2024GeoPos: A Minimal Positional Encoding for Enhanced Fine-Grained Details in Image Synthesis Using Convolutional Neural Networkscited by 1
  2. 2024LaFAM: Unsupervised Feature Attribution with Label-free Activation Mapscited by 1
  3. 2024Structured Event Reasoning with Large Language Modelscited by 0
  4. 2025CCNeXt: An Effective Self-Supervised Stereo Depth Estimation Approachcited by 0