# A Recent Survey of Vision Transformers for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2312.00634 · HTML: https://mlsanity.com/paper/2312.00634/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 186 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019)
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](https://mlsanity.com/paper/2109.03201/) (2021)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation](https://mlsanity.com/paper/2107.00781/) (2021)
- [ReSTR: Convolution-free Referring Image Segmentation Using Transformers](https://mlsanity.com/paper/2203.16768/) (2022)
- [ViTGAN: Training GANs with Vision Transformers](https://mlsanity.com/paper/2107.04589/) (2021)
- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [Deep Learning for Medical Image Processing: Overview, Challenges and Future](https://mlsanity.com/paper/1704.06825/) (2017)

## Built on by (most cited first, 23 of 23)

- [Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer](https://mlsanity.com/paper/2503.00226/) (2025)
- [Fit Pixels, Get Labels: Meta-learned Implicit Networks for Image Segmentation](https://mlsanity.com/paper/2510.04021/) (2025)
- [A Survey on Self-supervised Contrastive Learning for Multimodal Text-Image Analysis](https://mlsanity.com/paper/2503.11101/) (2025)
- [MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation](https://mlsanity.com/paper/2512.17774/) (2025)
- [RS-FME-SwinT: A Novel Feature Map Enhancement Framework Integrating Customized SwinT with Residual and Spatial CNN for Monkeypox Diagnosis](https://mlsanity.com/paper/2410.01216/) (2024)
- [A Novel Channel Boosted Residual CNN-Transformer with Regional-Boundary Learning for Breast Cancer Detection](https://mlsanity.com/paper/2503.15008/) (2025)
- [CE-RS-SBCIT A Novel Channel Enhanced Hybrid CNN Transformer with Residual, Spatial, and Boundary-Aware Learning for Brain Tumor MRI Analysis](https://mlsanity.com/paper/2508.17128/) (2025)
- [RS-CA-HSICT: A Residual and Spatial Channel Augmented CNN Transformer Framework for Monkeypox Detection](https://mlsanity.com/paper/2511.15476/) (2025)
- [AutoGen Driven Multi Agent Framework for Iterative Crime Data Analysis and Prediction](https://mlsanity.com/paper/2506.11475/) (2025)
- [Residual-SwinCA-Net: A Channel-Aware Integrated Residual CNN-Swin Transformer for Malignant Lesion Segmentation in BUSI](https://mlsanity.com/paper/2512.08243/) (2025)
- [Evaluation of Vision Transformers for Multimodal Image Classification: A Case Study on Brain, Lung, and Kidney Tumors](https://mlsanity.com/paper/2502.05517/) (2025)
- [Advanced Hybrid Transformer LSTM Technique with Attention and TS Mixer for Drilling Rate of Penetration Prediction](https://mlsanity.com/paper/2508.05210/) (2025)
- [VL-OrdinalFormer: Vision Language Guided Ordinal Transformers for Interpretable Knee Osteoarthritis Grading](https://mlsanity.com/paper/2601.00879/) (2025)
- [RSwinV2-MD: An Enhanced Residual SwinV2 Transformer for Monkeypox Detection from Skin Images](https://mlsanity.com/paper/2601.01835/) (2026)
- [T-DuMpRa: Teacher-guided Dual-path Multi-prototype Retrieval Augmented framework for fine-grained medical image classification](https://mlsanity.com/paper/2604.17360/) (2026)
- [A Novel Decision Ensemble Framework: Customized Attention-BiLSTM and XGBoost for Speculative Stock Price Forecasting](https://mlsanity.com/paper/2401.11621/) (2024)
- [Particle Multi-Axis Transformer for Jet Tagging](https://mlsanity.com/paper/2406.06638/) (2024)
- [VerteNet -- A Multi-Context Hybrid CNN Transformer for Accurate Vertebral Landmark Localization in Lateral Spine DXA Images](https://mlsanity.com/paper/2502.02097/) (2025)
- [Automated MRI Tumor Segmentation using hybrid U-Net with Transformer and Efficient Attention](https://mlsanity.com/paper/2506.15562/) (2025)
- [Are Vision Transformer Representations Semantically Meaningful? A Case Study in Medical Imaging](https://mlsanity.com/paper/2507.01788/) (2025)
- [Attention via Synaptic Plasticity is All You Need: A Biologically Inspired Spiking Neuromorphic Transformer](https://mlsanity.com/paper/2511.14691/) (2025)
- [Token-UNet: A New Case for Transformers Integration in Efficient and Interpretable 3D UNets for Brain Imaging Segmentation](https://mlsanity.com/paper/2602.20008/) (2026)
- [Learning Features, Not Layouts: Feature-Axis Attention as a Transfer Prior for Medical Imaging](https://mlsanity.com/paper/2610.13012/) (2026)

## Abstract (arXiv)

Medical image segmentation plays a crucial role in various healthcare applications, enabling accurate diagnosis, treatment planning, and disease monitoring. Traditionally, convolutional neural networks (CNNs) dominated this domain, excelling at local feature extraction. However, their limitations in capturing long-range dependencies across image regions pose challenges for segmenting complex, interconnected structures often encountered in medical data. In recent years, Vision Transformers (ViTs) have emerged as a promising technique for addressing the challenges in medical image segmentation. Their multi-scale attention mechanism enables effective modeling of long-range dependencies between distant structures, crucial for segmenting organs or lesions spanning the image. Additionally, ViTs' ability to discern subtle pattern heterogeneity allows for the precise delineation of intricate boundaries and edges, a critical aspect of accurate medical image segmentation. However, they do lack image-related inductive bias and translational invariance, potentially impacting their performance. Recently, researchers have come up with various ViT-based approaches that incorporate CNNs in their architectures, known as Hybrid Vision Transformers (HVTs) to capture local correlation in addition to the global information in the images. This survey paper provides a detailed review of the recent advancements in ViTs and HVTs for medical image segmentation. Along with the categorization of ViT and HVT-based medical image segmentation approaches, we also present a detailed overview of their real-time applications in several medical image modalities. This survey may serve as a valuable resource for researchers, healthcare practitioners, and students in understanding the state-of-the-art approaches for ViT-based medical image segmentation.