mlsanityData to 7 Oct 2026

Is it enough to optimize CNN architectures on ImageNet?

arXiv 2103.09108 ↗First posted 16 Mar 2021cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Classification performance based on ImageNet is the de-facto standard metric for CNN development. In this work we challenge the notion that CNN architecture design solely based on ImageNet leads to generally effective convolutional neural network (CNN) architectures that perform well on a diverse set of datasets and application domains. To this end, we investigate and ultimately improve ImageNet as a basis for deriving such architectures. We conduct an extensive empirical study for which we train 500 CNN architectures, sampled from the broad AnyNetX design space, on ImageNet as well as 8 additional well known image classification benchmark datasets from a diverse array of application domains. We observe that the performances of the architectures are highly dataset dependent. Some datasets even exhibit a negative error correlation with ImageNet across all architectures. We show how to significantly increase these correlations by utilizing ImageNet subsets restricted to fewer classes. These contributions can have a profound impact on the way we design future CNN architectures and help alleviate the tilt we see currently in our community with respect to over-reliance on one dataset.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 16 Mar 2021.

Builds on

34 of 53 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
22 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  4. 2020Designing Network Design Spacescited by 954
  5. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  6. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  7. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  8. 2018Do Better ImageNet Models Transfer Better?cited by 560
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  12. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  13. 2015LSTM: A Search Space Odysseycited by 315
  14. 2015Highway Networkscited by 257
  15. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  16. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  17. 2017Generalization in Deep Learningcited by 146
  18. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  19. 2019DetNAS: Backbone Search for Object Detectioncited by 80
  20. 2019On Network Design Spaces for Visual Recognitioncited by 68
  21. 2016Capacity and Trainability in Recurrent Neural Networkscited by 26
  22. 2018Deep Learning in the Wildcited by 12

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 3; 2024: 0; 2025: 1.

9 papers.

  1. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  2. 2023A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directionscited by 22
  3. 2021Tune It or Don't Use It: Benchmarking Data-Efficient Image Classificationcited by 11
  4. 2022Image Classification with Small Datasets: Overview and Benchmarkcited by 4
  5. 2021FEAR: A Simple Lightweight Method to Rank Architecturescited by 3
  6. 2022One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learningcited by 1
  7. 2022Analysis of Deep Learning Architectures and Efficacy of Detecting Forest Firescited by 0
  8. 2023Deep Neural Networks for Automatic Speaker Recognition Do Not Learn Supra-Segmental Temporal Featurescited by 0
  9. 2025Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Modelscited by 0