# An Impartial Take to the CNN vs Transformer Robustness Contest

arXiv: https://arxiv.org/abs/2207.11347 · HTML: https://mlsanity.com/paper/2207.11347/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Vision Transformers are Robust Learners](https://mlsanity.com/paper/2105.07581/) (2021)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Delving Deep into the Generalization of Vision Transformers under Distribution Shifts](https://mlsanity.com/paper/2106.07617/) (2021)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [Stable Rank Normalization for Improved Generalization in Neural Networks and GANs](https://mlsanity.com/paper/1906.04659/) (2019)
- [Performance measures for classification systems with rejection](https://mlsanity.com/paper/1504.02763/) (2015)

## Built on by (most cited first, 30 of 34)

- [A Light Recipe to Train Robust Vision Transformers](https://mlsanity.com/paper/2209.07399/) (2022)
- [MogaNet: Multi-order Gated Aggregation Network](https://mlsanity.com/paper/2211.03295/) (2022)
- [ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets](https://mlsanity.com/paper/2209.00613/) (2022)
- [Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models](https://mlsanity.com/paper/2306.02080/) (2023)
- [What do neural networks learn in image classification? A frequency shortcut perspective](https://mlsanity.com/paper/2307.09829/) (2023)
- [Anatomy-aware and acquisition-agnostic joint registration with SynthMorph](https://mlsanity.com/paper/2301.11329/) (2023)
- [ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy](https://mlsanity.com/paper/2311.09215/) (2023)
- [Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators](https://mlsanity.com/paper/2212.11237/) (2022)
- [MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection](https://mlsanity.com/paper/2309.14976/) (2023)
- [Reliability in Semantic Segmentation: Are We on the Right Track?](https://mlsanity.com/paper/2303.11298/) (2023)
- [Reliable Evaluation of Adversarial Transferability](https://mlsanity.com/paper/2306.08565/) (2023)
- [On the unreasonable vulnerability of transformers for image restoration -- and an easy fix](https://mlsanity.com/paper/2307.13856/) (2023)
- [Are We Ready for Out-of-Distribution Detection in Digital Pathology?](https://mlsanity.com/paper/2407.13708/) (2024)
- [E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition](https://mlsanity.com/paper/2305.17854/) (2023)
- [An Analysis of Initial Training Strategies for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2308.11677/) (2023)
- [A Rate-Distortion View of Uncertainty Quantification](https://mlsanity.com/paper/2406.10775/) (2024)
- [SkyGPT: Probabilistic Short-term Solar Forecasting Using Synthetic Sky Videos from Physics-constrained VideoGPT](https://mlsanity.com/paper/2306.11682/) (2023)
- [Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling](https://mlsanity.com/paper/2405.17139/) (2024)
- [Situation Monitor: Diversity-Driven Zero-Shot Out-of-Distribution Detection using Budding Ensemble Architecture for Object Detection](https://mlsanity.com/paper/2406.03188/) (2024)
- [On Evaluating Adversarial Robustness of Volumetric Medical Segmentation Models](https://mlsanity.com/paper/2406.08486/) (2024)
- [Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?](https://mlsanity.com/paper/2510.16060/) (2025)
- [Underwater target detection based on improved YOLOv7](https://mlsanity.com/paper/2302.06939/) (2023)
- [Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing](https://mlsanity.com/paper/2306.17848/) (2023)
- [Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks](https://mlsanity.com/paper/2309.06438/) (2023)
- [GeoPos: A Minimal Positional Encoding for Enhanced Fine-Grained Details in Image Synthesis Using Convolutional Neural Networks](https://mlsanity.com/paper/2401.01951/) (2024)
- [Adversarial Robustness of Discriminative Self-Supervised Learning in Vision](https://mlsanity.com/paper/2503.06361/) (2025)
- [Fine-Grained ImageNet Classification in the Wild](https://mlsanity.com/paper/2303.02400/) (2023)
- [Improving Stain Invariance of CNNs for Segmentation by Fusing Channel Attention and Domain-Adversarial Training](https://mlsanity.com/paper/2304.11445/) (2023)
- [A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention](https://mlsanity.com/paper/2407.18673/) (2024)
- [Divide and Conquer Self-Supervised Learning for High-Content Imaging](https://mlsanity.com/paper/2503.07444/) (2025)

## Abstract (arXiv)

Following the surge of popularity of Transformers in Computer Vision, several studies have attempted to determine whether they could be more robust to distribution shifts and provide better uncertainty estimates than Convolutional Neural Networks (CNNs). The almost unanimous conclusion is that they are, and it is often conjectured more or less explicitly that the reason of this supposed superiority is to be attributed to the self-attention mechanism. In this paper we perform extensive empirical analyses showing that recent state-of-the-art CNNs (particularly, ConvNeXt) can be as robust and reliable or even sometimes more than the current state-of-the-art Transformers. However, there is no clear winner. Therefore, although it is tempting to state the definitive superiority of one family of architectures over another, they seem to enjoy similar extraordinary performances on a variety of tasks while also suffering from similar vulnerabilities such as texture, background, and simplicity biases.