# From CNN to Transformer: A Review of Medical Image Segmentation Models

arXiv: https://arxiv.org/abs/2308.05305 · HTML: https://mlsanity.com/paper/2308.05305/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks](https://mlsanity.com/paper/1702.05970/) (2017)
- [Hough-CNN: Deep Learning for Segmentation of Deep Brain Regions in MRI and Ultrasound](https://mlsanity.com/paper/1601.07014/) (2016)

## Built on by (most cited first, 22 of 22)

- [SPINEPS -- Automatic Whole Spine Segmentation of T2-weighted MR images using a Two-Phase Approach to Multi-class Semantic and Instance Segmentation](https://mlsanity.com/paper/2402.16368/) (2024)
- [CAMS: Convolution and Attention-Free Mamba-based Cardiac Image Segmentation](https://mlsanity.com/paper/2406.05786/) (2024)
- [Diffusion Model as a Generalist Segmentation Learner](https://mlsanity.com/paper/2604.24575/) (2026)
- [Architecture Analysis and Benchmarking of 3D U-shaped Deep Learning Models for Thoracic Anatomical Segmentation](https://mlsanity.com/paper/2402.03230/) (2024)
- [Benchmarking Retinal Blood Vessel Segmentation Models for Cross-Dataset and Cross-Disease Generalization](https://mlsanity.com/paper/2406.14994/) (2024)
- [GeloVec: Higher Dimensional Geometric Smoothing for Coherent Visual Feature Extraction in Image Segmentation](https://mlsanity.com/paper/2505.01057/) (2025)
- [SAMRI: Segment Any MRI](https://mlsanity.com/paper/2510.26635/) (2025)
- [MSA-UNet3+: Multi-Scale Attention UNet3+ with New Supervised Prototypical Contrastive Loss for Coronary DSA Image Segmentation](https://mlsanity.com/paper/2504.05184/) (2025)
- [CARE: Anti-entanglement Ultrasound Image Segmentation via Channel-Aware Region Extrication](https://mlsanity.com/paper/2508.13899/) (2025)
- [Three-dimensional visualization of X-ray micro-CT with large-scale datasets: Efficiency and accuracy for real-time interaction](https://mlsanity.com/paper/2601.15098/) (2026)
- [Judging from Support-set: A New Way to Utilize Few-Shot Segmentation for Segmentation Refinement Process](https://mlsanity.com/paper/2407.04519/) (2024)
- [Nd-BiMamba2: A Unified Bidirectional Architecture for Multi-Dimensional Data Processing](https://mlsanity.com/paper/2411.15380/) (2024)
- [Large model retrieval enhancement framework for construction site risk identification](https://mlsanity.com/paper/2508.02073/) (2025)
- [Improved Segmentation of Polyps and Visual Explainability Analysis](https://mlsanity.com/paper/2509.18159/) (2025)
- [Comparative Study of UNet-based Architectures for Liver Tumor Segmentation in Multi-Phase Contrast-Enhanced Computed Tomography](https://mlsanity.com/paper/2510.25522/) (2025)
- [Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2603.06374/) (2026)
- [Decoding Matters: Efficient Mamba-Based Decoder with Distribution-Aware Deep Supervision for Medical Image Segmentation](https://mlsanity.com/paper/2603.12547/) (2026)
- [Exploring the Impact of Skin Color on Skin Lesion Segmentation](https://mlsanity.com/paper/2603.29694/) (2026)
- [Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detection](https://mlsanity.com/paper/2605.01563/) (2026)
- [AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation](https://mlsanity.com/paper/2606.07633/) (2026)
- [ESRVS: Extreme Semi-Supervised Retinal Vessel Segmentation with a Single Annotated Image](https://mlsanity.com/paper/2607.24453/) (2026)
- [Enhancing Low Back Pain Assessment with Diffusion Models for Lumbar Spine MRI Segmentation](https://mlsanity.com/paper/2608.04906/) (2026)

## Abstract (arXiv)

Medical image segmentation is an important step in medical image analysis, especially as a crucial prerequisite for efficient disease diagnosis and treatment. The use of deep learning for image segmentation has become a prevalent trend. The widely adopted approach currently is U-Net and its variants. Additionally, with the remarkable success of pre-trained models in natural language processing tasks, transformer-based models like TransUNet have achieved desirable performance on multiple medical image segmentation datasets. In this paper, we conduct a survey of the most representative four medical image segmentation models in recent years. We theoretically analyze the characteristics of these models and quantitatively evaluate their performance on two benchmark datasets (i.e., Tuberculosis Chest X-rays and ovarian tumors). Finally, we discuss the main challenges and future trends in medical image segmentation. Our work can assist researchers in the related field to quickly establish medical segmentation models tailored to specific regions.