# A comparative study between vision transformers and CNNs in digital pathology

arXiv: https://arxiv.org/abs/2206.00389 · HTML: https://mlsanity.com/paper/2206.00389/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Automated Classification of Histopathology Images Using Transfer Learning](https://mlsanity.com/paper/1903.10035/) (2019)
- [Understanding the Mechanisms of Deep Transfer Learning for Medical Images](https://mlsanity.com/paper/1704.06040/) (2017)

## Built on by (most cited first, 16 of 16)

- [Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey](https://mlsanity.com/paper/2402.02941/) (2024)
- [Towards Natural Machine Unlearning](https://mlsanity.com/paper/2405.15495/) (2024)
- [Evaluating histopathology transfer learning with ChampKit](https://mlsanity.com/paper/2206.06862/) (2022)
- [Large-scale pretraining on pathological images for fine-tuning of small pathological benchmarks](https://mlsanity.com/paper/2303.15693/) (2023)
- [VyAnG-Net: A Novel Multi-Modal Sarcasm Recognition Model by Uncovering Visual, Acoustic and Glossary Features](https://mlsanity.com/paper/2408.10246/) (2024)
- [FLARES: Fast and Accurate LiDAR Multi-Range Semantic Segmentation](https://mlsanity.com/paper/2502.09274/) (2025)
- [PoCo: Point Context Cluster for RGBD Indoor Place Recognition](https://mlsanity.com/paper/2404.02885/) (2024)
- [Convolutional Neural Networks and Vision Transformers for Fashion MNIST Classification: A Literature Review](https://mlsanity.com/paper/2406.03478/) (2024)
- [Frequency-Based Vulnerability Analysis of Deep Learning Models against Image Corruptions](https://mlsanity.com/paper/2306.07178/) (2023)
- [On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery](https://mlsanity.com/paper/2409.12026/) (2024)
- [Foundations and Models in Modern Computer Vision: Key Building Blocks in Landmark Architectures](https://mlsanity.com/paper/2507.23357/) (2025)
- [Classification in Histopathology: A unique deep embeddings extractor for multiple classification tasks](https://mlsanity.com/paper/2303.05180/) (2023)
- [Glioma subtype classification from histopathological images using in-domain and out-of-domain transfer learning: An experimental study](https://mlsanity.com/paper/2309.17223/) (2023)
- [Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoring](https://mlsanity.com/paper/2407.13341/) (2024)
- [Are Vision Transformer Representations Semantically Meaningful? A Case Study in Medical Imaging](https://mlsanity.com/paper/2507.01788/) (2025)
- [Enhancing Shrimp Disease Detection via Deep Learning and Data Refinement for Resilient Aquaculture](https://mlsanity.com/paper/2609.23397/) (2026)

## Abstract (arXiv)

Recently, vision transformers were shown to be capable of outperforming convolutional neural networks when pretrained on sufficient amounts of data. In comparison to convolutional neural networks, vision transformers have a weaker inductive bias and therefore allow a more flexible feature detection. Due to their promising feature detection, this work explores vision transformers for tumor detection in digital pathology whole slide images in four tissue types, and for tissue type identification. We compared the patch-wise classification performance of the vision transformer DeiT-Tiny to the state-of-the-art convolutional neural network ResNet18. Due to the sparse availability of annotated whole slide images, we further compared both models pretrained on large amounts of unlabeled whole-slide images using state-of-the-art self-supervised approaches. The results show that the vision transformer performed slightly better than the ResNet18 for three of four tissue types for tumor detection while the ResNet18 performed slightly better for the remaining tasks. The aggregated predictions of both models on slide level were correlated, indicating that the models captured similar imaging features. All together, the vision transformer models performed on par with the ResNet18 while requiring more effort to train. In order to surpass the performance of convolutional neural networks, vision transformers might require more challenging tasks to benefit from their weak inductive bias.