# ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy

arXiv: https://arxiv.org/abs/2311.09215 · HTML: https://mlsanity.com/paper/2311.09215/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 65 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)

## Built on by (most cited first, 19 of 19)

- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks](https://mlsanity.com/paper/2403.00522/) (2024)
- [ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models](https://mlsanity.com/paper/2405.15738/) (2024)
- [Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://mlsanity.com/paper/2412.08737/) (2024)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training](https://mlsanity.com/paper/2403.08109/) (2024)
- [An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Training](https://mlsanity.com/paper/2404.12210/) (2024)
- [Adapting LLaMA Decoder to Vision Transformer](https://mlsanity.com/paper/2404.06773/) (2024)
- [Deep learning-based ecological analysis of camera trap images is impacted by training data quality and quantity](https://mlsanity.com/paper/2408.14348/) (2024)
- [Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning](https://mlsanity.com/paper/2410.06373/) (2024)
- [Vision-Language Model Selection and Reuse for Downstream Adaptation](https://mlsanity.com/paper/2501.18271/) (2025)
- [Vision Backbone Efficient Selection for Image Classification in Low-Data Regimes](https://mlsanity.com/paper/2410.08592/) (2024)
- [Image Recognition with Vision and Language Embeddings of VLMs](https://mlsanity.com/paper/2509.09311/) (2025)
- [Intriguing Properties of Large Language and Vision Models](https://mlsanity.com/paper/2410.04751/) (2024)
- [Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Models](https://mlsanity.com/paper/2511.00335/) (2025)
- [BioBench: A Blueprint to Move Beyond ImageNet for Scientific ML Benchmarks](https://mlsanity.com/paper/2511.16315/) (2025)
- [Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study](https://mlsanity.com/paper/2605.15599/) (2026)
- [MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification](https://mlsanity.com/paper/2608.13463/) (2026)

## Abstract (arXiv)

Modern computer vision offers a great variety of models to practitioners, and selecting a model from multiple options for specific applications can be challenging. Conventionally, competing model architectures and training protocols are compared by their classification accuracy on ImageNet. However, this single metric does not fully capture performance nuances critical for specialized tasks. In this work, we conduct an in-depth comparative analysis of model behaviors beyond ImageNet accuracy, for both ConvNet and Vision Transformer architectures, each across supervised and CLIP training paradigms. Although our selected models have similar ImageNet accuracies and compute requirements, we find that they differ in many other aspects: types of mistakes, output calibration, transferability, and feature invariance, among others. This diversity in model characteristics, not captured by traditional metrics, highlights the need for more nuanced analysis when choosing among different models. Our code is available at https://github.com/kirill-vish/Beyond-INet.