# Vision Transformer for Efficient Chest X-ray and Gastrointestinal Image Classification

arXiv: https://arxiv.org/abs/2304.11529 · HTML: https://mlsanity.com/paper/2304.11529/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 23 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation](https://mlsanity.com/paper/1802.06955/) (2018)
- [COVID-19 Image Data Collection](https://mlsanity.com/paper/2003.11597/) (2020)
- [Is it Time to Replace CNNs with Transformers for Medical Images?](https://mlsanity.com/paper/2108.09038/) (2021)
- [Reliable Tuberculosis Detection using Chest X-ray with Deep Learning, Segmentation and Visualization](https://mlsanity.com/paper/2007.14895/) (2020)
- [Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms](https://mlsanity.com/paper/1910.07387/) (2019)
- [VORTEX: Physics-Driven Data Augmentations Using Consistency Training for Robust Accelerated MRI Reconstruction](https://mlsanity.com/paper/2111.02549/) (2021)
- [Deep Learning for Reliable Classification of COVID-19, MERS, and SARS from Chest X-Ray Images](https://mlsanity.com/paper/2005.11524/) (2020)
- [Video Capsule Endoscopy Classification using Focal Modulation Guided Convolutional Neural Network](https://mlsanity.com/paper/2206.08298/) (2022)
- [The Medico-Task 2018: Disease Detection in the Gastrointestinal Tract using Global Features and Deep Learning](https://mlsanity.com/paper/1810.13278/) (2018)
- [Gastrointestinal Disorder Detection with a Transformer Based Approach](https://mlsanity.com/paper/2210.03168/) (2022)

## Built on by (most cited first, 2 of 2)

- [MedBlindTuner: Towards Privacy-preserving Fine-tuning on Biomedical Images with Transformers and Fully Homomorphic Encryption](https://mlsanity.com/paper/2401.09604/) (2024)
- [Local Lesion Generation is Effective for Capsule Endoscopy Image Data Augmentation in a Limited Data Setting](https://mlsanity.com/paper/2411.03098/) (2024)

## Abstract (arXiv)

Medical image analysis is a hot research topic because of its usefulness in different clinical applications, such as early disease diagnosis and treatment. Convolutional neural networks (CNNs) have become the de-facto standard in medical image analysis tasks because of their ability to learn complex features from the available datasets, which makes them surpass humans in many image-understanding tasks. In addition to CNNs, transformer architectures also have gained popularity for medical image analysis tasks. However, despite progress in the field, there are still potential areas for improvement. This study uses different CNNs and transformer-based methods with a wide range of data augmentation techniques. We evaluated their performance on three medical image datasets from different modalities. We evaluated and compared the performance of the vision transformer model with other state-of-the-art (SOTA) pre-trained CNN networks. For Chest X-ray, our vision transformer model achieved the highest F1 score of 0.9532, recall of 0.9533, Matthews correlation coefficient (MCC) of 0.9259, and ROC-AUC score of 0.97. Similarly, for the Kvasir dataset, we achieved an F1 score of 0.9436, recall of 0.9437, MCC of 0.9360, and ROC-AUC score of 0.97. For the Kvasir-Capsule (a large-scale VCE dataset), our ViT model achieved a weighted F1-score of 0.7156, recall of 0.7182, MCC of 0.3705, and ROC-AUC score of 0.57. We found that our transformer-based models were better or more effective than various CNN models for classifying different anatomical structures, findings, and abnormalities. Our model showed improvement over the CNN-based approaches and suggests that it could be used as a new benchmarking algorithm for algorithm development.