# Is it Time to Replace CNNs with Transformers for Medical Images?

arXiv: https://arxiv.org/abs/2108.09038 · HTML: https://mlsanity.com/paper/2108.09038/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration](https://mlsanity.com/paper/2104.06468/) (2021)

## Built on by (most cited first, 30 of 60)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification](https://mlsanity.com/paper/2210.12843/) (2022)
- [What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors](https://mlsanity.com/paper/2203.01825/) (2022)
- [Swin Transformer for Fast MRI](https://mlsanity.com/paper/2201.03230/) (2022)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [PatchDropout: Economizing Vision Transformers Using Patch Dropout](https://mlsanity.com/paper/2208.07220/) (2022)
- [Large-scale Long-tailed Disease Diagnosis on Radiology Images](https://mlsanity.com/paper/2312.16151/) (2023)
- [Multi-modal Deep Learning](https://mlsanity.com/paper/2403.03385/) (2024)
- [Pretrained ViTs Yield Versatile Representations For Medical Images](https://mlsanity.com/paper/2303.07034/) (2023)
- [Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms](https://mlsanity.com/paper/2206.10096/) (2022)
- [Vision transformers in domain adaptation and domain generalization: a study of robustness](https://mlsanity.com/paper/2404.04452/) (2024)
- [Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labels](https://mlsanity.com/paper/2209.07819/) (2022)
- [SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2210.10969/) (2022)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [Metadata-guided Consistency Learning for High Content Images](https://mlsanity.com/paper/2212.11595/) (2022)
- [Fast MRI Reconstruction: How Powerful Transformers Are?](https://mlsanity.com/paper/2201.09400/) (2022)
- [Efficiently Training Vision Transformers on Structural MRI Scans for Alzheimer's Disease Detection](https://mlsanity.com/paper/2303.08216/) (2023)
- [Evaluating Visual Explanations of Attention Maps for Transformer-based Medical Imaging](https://mlsanity.com/paper/2503.09535/) (2025)
- [Automatic Segmentation of Head and Neck Tumor: How Powerful Transformers Are?](https://mlsanity.com/paper/2201.06251/) (2022)
- [CASS: Cross Architectural Self-Supervision for Medical Image Analysis](https://mlsanity.com/paper/2206.04170/) (2022)
- [SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Images](https://mlsanity.com/paper/2302.03861/) (2023)
- [Transformer-Based Approaches for Sensor-Based Human Activity Recognition: Opportunities and Challenges](https://mlsanity.com/paper/2410.13605/) (2024)
- [How Well Do Vision Transformers (VTs) Transfer To The Non-Natural Image Domain? An Empirical Study Involving Art Classification](https://mlsanity.com/paper/2208.04693/) (2022)
- [MultiNet with Transformers: A Model for Cancer Diagnosis Using Images](https://mlsanity.com/paper/2301.09007/) (2023)
- [TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction](https://mlsanity.com/paper/2301.10829/) (2023)
- [MGMT promoter methylation status prediction using MRI scans? An extensive experimental evaluation of deep learning models](https://mlsanity.com/paper/2304.00774/) (2023)
- [FissionFusion: Fast Geometric Generation and Hierarchical Souping for Medical Image Analysis](https://mlsanity.com/paper/2403.13341/) (2024)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis. Recently, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding similar levels of performance while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is time to move to transformer-based models or if we should keep working with CNNs - can we trivially switch to transformers? If so, what are the advantages and drawbacks of switching to ViTs for medical image diagnosis? We consider these questions in a series of experiments on three mainstream medical image datasets. Our findings show that, while CNNs perform better when trained from scratch, off-the-shelf vision transformers using default hyperparameters are on par with CNNs when pretrained on ImageNet, and outperform their CNN counterparts when pretrained using self-supervision.