mlsanityData to 7 Oct 2026

Is it Time to Replace CNNs with Transformers for Medical Images?

arXiv 2108.09038 ↗First posted 20 Aug 2021cs.CV cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis. Recently, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding similar levels of performance while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is time to move to transformer-based models or if we should keep working with CNNs - can we trivially switch to transformers? If so, what are the advantages and drawbacks of switching to ViTs for medical image diagnosis? We consider these questions in a series of experiments on three mainstream medical image datasets. Our findings show that, while CNNs perform better when trained from scratch, off-the-shelf vision transformers using default hyperparameters are on par with CNNs when pretrained on ImageNet, and outperform their CNN counterparts when pretrained using self-supervision.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 20 Aug 2021.

Builds on

18 of 26 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  12. 2021Vision Transformers for Dense Predictioncited by 1,789
6 more
  1. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  4. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  5. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  6. 2021ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registrationcited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 21; 2023: 17; 2024: 12; 2025: 5; 2026: 3.

60 papers.

Sort:

By area:

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  3. 2022What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factorscited by 61
  4. 2022Swin Transformer for Fast MRIcited by 56
  5. 2023Are Natural Domain Foundation Models Useful for Medical Image Classification?cited by 44
  6. 2022Transformers in Medical Image Analysis: A Reviewcited by 41
  7. 2022Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectivescited by 32
  8. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 31
  9. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  10. 2023Large-scale Long-tailed Disease Diagnosis on Radiology Imagescited by 17
  11. 2024Multi-modal Deep Learningcited by 15
  12. 2023Pretrained ViTs Yield Versatile Representations For Medical Imagescited by 13
38 more
  1. 2022Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammogramscited by 11
  2. 2024Vision transformers in domain adaptation and domain generalization: a study of robustnesscited by 11
  3. 2022Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labelscited by 9
  4. 2022SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Gradingcited by 9
  5. 2022Vision Transformers in Medical Imaging: A Reviewcited by 8
  6. 2022Metadata-guided Consistency Learning for High Content Imagescited by 8
  7. 2022Fast MRI Reconstruction: How Powerful Transformers Are?cited by 7
  8. 2023Efficiently Training Vision Transformers on Structural MRI Scans for Alzheimer's Disease Detectioncited by 7
  9. 2025Evaluating Visual Explanations of Attention Maps for Transformer-based Medical Imagingcited by 7
  10. 2022Automatic Segmentation of Head and Neck Tumor: How Powerful Transformers Are?cited by 6
  11. 2022CASS: Cross Architectural Self-Supervision for Medical Image Analysiscited by 6
  12. 2023SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Imagescited by 5
  13. 2024Transformer-Based Approaches for Sensor-Based Human Activity Recognition: Opportunities and Challengescited by 4
  14. 2022How Well Do Vision Transformers (VTs) Transfer To The Non-Natural Image Domain? An Empirical Study Involving Art Classificationcited by 3
  15. 2023MultiNet with Transformers: A Model for Cancer Diagnosis Using Imagescited by 3
  16. 2023TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Predictioncited by 3
  17. 2023MGMT promoter methylation status prediction using MRI scans? An extensive experimental evaluation of deep learning modelscited by 3
  18. 2024FissionFusion: Fast Geometric Generation and Hierarchical Souping for Medical Image Analysiscited by 3
  19. 2024Evaluating the Efficacy of Prompt-Engineered Large Multimodal Models Versus Fine-Tuned Vision Transformers in Image-Based Security Applicationscited by 3
  20. 2024Sparsity- and Hybridity-Inspired Visual Parameter-Efficient Fine-Tuning for Medical Diagnosiscited by 3
  21. 2024Classification of Endoscopy and Video Capsule Images using CNN-Transformer Modelcited by 3
  22. 2025MedFormer: Hierarchical Medical Vision Transformer with Content-Aware Dual Sparse Selection Attentioncited by 3
  23. 2021Dispensed Transformer Network for Unsupervised Domain Adaptationcited by 2
  24. 2022Introducing Vision Transformer for Alzheimer's Disease classification task with 3D inputcited by 2
  25. 2023Vision Transformer for Efficient Chest X-ray and Gastrointestinal Image Classificationcited by 2
  26. 2023Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervisioncited by 2
  27. 2024Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impactcited by 2
  28. 2022Deep is a Luxury We Don't Havecited by 1
  29. 2022Masked autoencoders are effective solution to transformer data-hungrycited by 1
  30. 2022Investigation of Network Architecture for Multimodal Head-and-Neck Tumor Segmentationcited by 1
  31. 2023Self-Supervised Curricular Deep Learning for Chest X-Ray Image Classificationcited by 1
  32. 2023Connectional-Style-Guided Contextual Representation Learning for Brain Disease Diagnosiscited by 1
  33. 2023Hardwiring ViT Patch Selectivity into CNNs using Patch Mixingcited by 1
  34. 2023Video and Synthetic MRI Pre-training of 3D Vision Architectures for Neuroimage Analysiscited by 1
  35. 2025Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classificationcited by 1
  36. 2021RA V-Net: Deep learning network for automated liver segmentationcited by 0
  37. 2023Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learningcited by 0
  38. 2023A Comprehensive Study of Modern Architectures and Regularization Approaches on CheXpert5000cited by 0

50 most cited of 60 All 60 (JSON)