# Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

arXiv: https://arxiv.org/abs/2301.03505 · HTML: https://mlsanity.com/paper/2301.03505/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 128 of its 242 references that are in the map.
- First posted on arXiv 9 Jan 2023.

## Builds on (most cited first, 30 of 128)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)

## Built on by (most cited first, 30 of 62)

- [Implicit Neural Representation in Medical Imaging: A Comparative Survey](https://mlsanity.com/paper/2307.16142/) (2023)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation](https://mlsanity.com/paper/2309.00121/) (2023)
- [Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis](https://mlsanity.com/paper/2402.01730/) (2024)
- [A Recent Survey of Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2312.00634/) (2023)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging](https://mlsanity.com/paper/2402.17246/) (2024)
- [Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey](https://mlsanity.com/paper/2402.02941/) (2024)
- [DermoSegDiff: A Boundary-aware Segmentation Diffusion Model for Skin Lesion Delineation](https://mlsanity.com/paper/2308.02959/) (2023)
- [xLSTM-UNet can be an Effective 2D & 3D Medical Image Segmentation Backbone with Vision-LSTM (ViL) better than its Mamba Counterpart](https://mlsanity.com/paper/2407.01530/) (2024)
- [Enhancing Medical Image Segmentation with TransCeption: A Multi-Scale Feature Fusion Approach](https://mlsanity.com/paper/2301.10847/) (2023)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Circuit Complexity Bounds for Visual Autoregressive Model](https://mlsanity.com/paper/2501.04299/) (2025)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma](https://mlsanity.com/paper/2405.12963/) (2024)
- [Tissue Concepts: supervised foundation models in computational pathology](https://mlsanity.com/paper/2409.03519/) (2024)
- [Recent Advances in Medical Imaging Segmentation: A Survey](https://mlsanity.com/paper/2505.09274/) (2025)
- [Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth](https://mlsanity.com/paper/2502.20758/) (2025)
- [Self-supervised Few-shot Learning for Semantic Segmentation: An Annotation-free Approach](https://mlsanity.com/paper/2307.14446/) (2023)
- [Enhancing Answer Reliability Through Inter-Model Consensus of Large Language Models](https://mlsanity.com/paper/2411.16797/) (2024)
- [Self-supervised Semantic Segmentation: Consistency over Transformation](https://mlsanity.com/paper/2309.00143/) (2023)
- [Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention](https://mlsanity.com/paper/2502.13693/) (2025)
- [Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights](https://mlsanity.com/paper/2403.19882/) (2024)
- [LHU-Net: a Lean Hybrid U-Net for Cost-efficient, High-performance Volumetric Segmentation](https://mlsanity.com/paper/2404.05102/) (2024)
- [Deep Learning for Pancreas Segmentation: a Systematic Review](https://mlsanity.com/paper/2407.16313/) (2024)
- [MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation](https://mlsanity.com/paper/2410.02458/) (2024)
- [A Self-supervised Multimodal Deep Learning Approach to Differentiate Post-radiotherapy Progression from Pseudoprogression in Glioblastoma](https://mlsanity.com/paper/2502.03999/) (2025)
- [Meningioma Analysis and Diagnosis using Limited Labeled Samples](https://mlsanity.com/paper/2602.13335/) (2026)
- [Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers](https://mlsanity.com/paper/2308.13442/) (2023)

## Abstract (arXiv)

The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.