mlsanityData to 7 Oct 2026

ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy

arXiv 2311.09215 ↗First posted 15 Nov 2023cs.CV cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Modern computer vision offers a great variety of models to practitioners, and selecting a model from multiple options for specific applications can be challenging. Conventionally, competing model architectures and training protocols are compared by their classification accuracy on ImageNet. However, this single metric does not fully capture performance nuances critical for specialized tasks. In this work, we conduct an in-depth comparative analysis of model behaviors beyond ImageNet accuracy, for both ConvNet and Vision Transformer architectures, each across supervised and CLIP training paradigms. Although our selected models have similar ImageNet accuracies and compute requirements, we find that they differ in many other aspects: types of mistakes, output calibration, transferability, and feature invariance, among others. This diversity in model characteristics, not captured by traditional metrics, highlights the need for more nuanced analysis when choosing among different models. Our code is available at https://github.com/kirill-vish/Beyond-INet.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 61 of its 65 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

61 of 65 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2016Layer Normalizationcited by 5,579
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
38 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  9. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  13. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  14. 2018Do Better ImageNet Models Transfer Better?cited by 560
  15. 2021Intriguing Properties of Vision Transformerscited by 446
  16. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  17. 2022DeiT III: Revenge of the ViTcited by 417
  18. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  19. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  20. 2020Are we done with ImageNet?cited by 329
  21. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  22. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  23. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  24. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  25. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  26. 2021Are Transformers More Robust Than CNNs?cited by 187
  27. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  28. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  29. 2021Are Convolutional Neural Networks or Transformers more like human vision?cited by 109
  30. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  31. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  32. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  33. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  34. 2022CLIPascene: Scene Sketching with Different Types and Levels of Abstractioncited by 63
  35. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  36. 2022ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotationscited by 44
  37. 2022Can CNNs Be More Robust Than Transformers?cited by 38
  38. 2021ConvNets vs. Transformers: Whose Visual Representations are More Transferable?cited by 35

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 4; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28
  4. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  5. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 12
  6. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  7. 2024VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Trainingcited by 5
  8. 2024An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Trainingcited by 4
  9. 2024Adapting LLaMA Decoder to Vision Transformercited by 3
  10. 2024Deep learning-based ecological analysis of camera trap images is impacted by training data quality and quantitycited by 3
  11. 2025Vision-Language Model Selection and Reuse for Downstream Adaptationcited by 3
  12. 2024Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learningcited by 2
7 more
  1. 2024Vision Backbone Efficient Selection for Image Classification in Low-Data Regimescited by 2
  2. 2025Image Recognition with Vision and Language Embeddings of VLMscited by 2
  3. 2024Intriguing Properties of Large Language and Vision Modelscited by 1
  4. 2025Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Modelscited by 0
  5. 2025BioBench: A Blueprint to Move Beyond ImageNet for Scientific ML Benchmarkscited by 0
  6. 2026Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Studycited by 0
  7. 2026MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classificationcited by 0