# Transformers in Medical Image Analysis: A Review

arXiv: https://arxiv.org/abs/2202.12165 · HTML: https://mlsanity.com/paper/2202.12165/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 114 of its 220 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 30 of 114)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)

## Built on by (most cited first, 30 of 41)

- [Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis](https://mlsanity.com/paper/2305.18453/) (2023)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation](https://mlsanity.com/paper/2211.01784/) (2022)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning](https://mlsanity.com/paper/2206.14413/) (2022)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers](https://mlsanity.com/paper/2308.09372/) (2023)
- [Fully transformer-based biomarker prediction from colorectal cancer histology: a large-scale multicentric study](https://mlsanity.com/paper/2301.09617/) (2023)
- [Boosting multi-demographic federated learning for chest radiograph analysis using general-purpose self-supervised representations](https://mlsanity.com/paper/2504.08584/) (2025)
- [Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning](https://mlsanity.com/paper/2505.17436/) (2025)
- [Transformers-based architectures for stroke segmentation: A review](https://mlsanity.com/paper/2403.18637/) (2024)
- [Artificial Intelligence in Ovarian Cancer Histopathology: A Systematic Review](https://mlsanity.com/paper/2303.18005/) (2023)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Gradient Boosting Decision Trees on Medical Diagnosis over Tabular Data](https://mlsanity.com/paper/2410.03705/) (2024)
- [A Self-supervised Multimodal Deep Learning Approach to Differentiate Post-radiotherapy Progression from Pseudoprogression in Glioblastoma](https://mlsanity.com/paper/2502.03999/) (2025)
- [Robust Fairness Vision-Language Learning for Medical Image Analysis](https://mlsanity.com/paper/2505.03153/) (2025)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [Lesion-Aware Cross-Phase Attention Network for Renal Tumor Subtype Classification on Multi-Phase CT Scans](https://mlsanity.com/paper/2406.16322/) (2024)
- [Impact of imperfect annotations on CNN training and performance for instance segmentation and classification in digital pathology](https://mlsanity.com/paper/2410.14365/) (2024)
- [Automatic Diagnosis of Myocarditis Disease in Cardiac MRI Modality using Deep Transformers and Explainable Artificial Intelligence](https://mlsanity.com/paper/2210.14611/) (2022)
- [MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data](https://mlsanity.com/paper/2211.06726/) (2022)
- [Transformer-based end-to-end classification of variable-length volumetric data](https://mlsanity.com/paper/2307.06666/) (2023)
- [Improving diagnosis and prognosis of lung cancer using vision transformers: A scoping review](https://mlsanity.com/paper/2309.02783/) (2023)
- [Automatic hip osteoarthritis grading with uncertainty estimation from computed tomography using digitally-reconstructed radiographs](https://mlsanity.com/paper/2401.00159/) (2023)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [SQ-Swin: a Pretrained Siamese Quadratic Swin Transformer for Lettuce Browning Prediction](https://mlsanity.com/paper/2209.07683/) (2022)
- [A residual dense vision transformer for medical image super-resolution with segmentation-based perceptual loss fine-tuning](https://mlsanity.com/paper/2302.11184/) (2023)

## Abstract (arXiv)

Transformers have dominated the field of natural language processing, and recently impacted the computer vision area. In the field of medical image analysis, Transformers have also been successfully applied to full-stack clinical applications, including image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. Our paper aims to promote awareness and application of Transformers in the field of medical image analysis. Specifically, we first overview the core concepts of the attention mechanism built into Transformers and other basic components. Second, we review various Transformer architectures tailored for medical image applications and discuss their limitations. Within this review, we investigate key challenges revolving around the use of Transformers in different learning paradigms, improving the model efficiency, and their coupling with other techniques. We hope this review can give a comprehensive picture of Transformers to the readers in the field of medical image analysis.