# TransDeepLab: Convolution-Free Transformer-based DeepLab v3+ for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2208.00713 · HTML: https://mlsanity.com/paper/2208.00713/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2106.06716/) (2021)
- [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2207.08518/) (2022)
- [Medical Image Segmentation Using Squeeze-and-Expansion Transformers](https://mlsanity.com/paper/2105.09511/) (2021)
- [On the Texture Bias for Few-Shot CNN Segmentation](https://mlsanity.com/paper/2003.04052/) (2020)
- [Semi-supervised few-shot learning for medical image segmentation](https://mlsanity.com/paper/2003.08462/) (2020)
- [Contextual Attention Network: Transformer Meets U-Net](https://mlsanity.com/paper/2203.01932/) (2022)
- [SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities](https://mlsanity.com/paper/2204.02961/) (2022)
- [Multi-level Context Gating of Embedded Collective Knowledge for Medical Image Segmentation](https://mlsanity.com/paper/2003.05056/) (2020)
- [Medical Image Segmentation on MRI Images with Missing Modalities: A Review](https://mlsanity.com/paper/2203.06217/) (2022)

## Built on by (most cited first, 30 of 43)

- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding](https://mlsanity.com/paper/2403.18271/) (2024)
- [VM-UNET-V2 Rethinking Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2403.09157/) (2024)
- [SAM3D: Segment Anything Model in Volumetric Medical Images](https://mlsanity.com/paper/2309.03493/) (2023)
- [Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation](https://mlsanity.com/paper/2309.00121/) (2023)
- [DAE-Former: Dual Attention-guided Efficient Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2212.13504/) (2022)
- [A Recent Survey of Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2312.00634/) (2023)
- [Attention Swin U-Net: Cross-Contextual Attention Mechanism for Skin Lesion Segmentation](https://mlsanity.com/paper/2210.16898/) (2022)
- [Self-Sampling Meta SAM: Enhancing Few-shot Medical Image Segmentation with Meta-Learning](https://mlsanity.com/paper/2308.16466/) (2023)
- [SegStitch: Multidimensional Transformer for Robust and Efficient Medical Imaging Segmentation](https://mlsanity.com/paper/2408.00496/) (2024)
- [MedSegFactory: Text-Guided Generation of Medical Image-Mask Pairs](https://mlsanity.com/paper/2504.06897/) (2025)
- [Enhancing Medical Image Segmentation with TransCeption: A Multi-Scale Feature Fusion Approach](https://mlsanity.com/paper/2301.10847/) (2023)
- [SliceMamba with Neural Architecture Search for Medical Image Segmentation](https://mlsanity.com/paper/2407.08481/) (2024)
- [Laplacian-Former: Overcoming the Limitations of Vision Transformers in Local Texture Detection](https://mlsanity.com/paper/2309.00108/) (2023)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [RWKV-UNet: Improving UNet with Long-Range Cooperation for Effective Medical Image Segmentation](https://mlsanity.com/paper/2501.08458/) (2025)
- [TransUKAN:Computing-Efficient Hybrid KAN-Transformer for Enhanced Medical Image Segmentation](https://mlsanity.com/paper/2409.14676/) (2024)
- [Can Foundation Models Really Segment Tumors? A Benchmarking Odyssey in Lung CT Imaging](https://mlsanity.com/paper/2505.01239/) (2025)
- [Merging Context Clustering with Visual State Space Models for Medical Image Segmentation](https://mlsanity.com/paper/2501.01618/) (2025)
- [Scale-aware Test-time Click Adaptation for Pulmonary Nodule and Mass Segmentation](https://mlsanity.com/paper/2307.15645/) (2023)
- [Medical Image Segmentation via Sparse Coding Decoder](https://mlsanity.com/paper/2310.10957/) (2023)
- [Deep models for stroke segmentation: do complex architectures always perform better?](https://mlsanity.com/paper/2403.17177/) (2024)
- [AgileFormer: Spatially Agile Transformer UNet for Medical Image Segmentation](https://mlsanity.com/paper/2404.00122/) (2024)
- [PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation](https://mlsanity.com/paper/2503.18227/) (2025)
- [Transfer Learning for Microstructure Segmentation with CS-UNet: A Hybrid Algorithm with Transformer and CNN Encoders](https://mlsanity.com/paper/2308.13917/) (2023)
- [SACNet: A Spatially Adaptive Convolution Network for 2D Multi-organ Medical Segmentation](https://mlsanity.com/paper/2407.10157/) (2024)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have been the de facto standard in a diverse set of computer vision tasks for many years. Especially, deep neural networks based on seminal architectures such as U-shaped models with skip-connections or atrous convolution with pyramid pooling have been tailored to a wide range of medical image analysis tasks. The main advantage of such architectures is that they are prone to detaining versatile local features. However, as a general consensus, CNNs fail to capture long-range dependencies and spatial correlations due to the intrinsic property of confined receptive field size of convolution operations. Alternatively, Transformer, profiting from global information modelling that stems from the self-attention mechanism, has recently attained remarkable performance in natural language processing and computer vision. Nevertheless, previous studies prove that both local and global features are critical for a deep model in dense prediction, such as segmenting complicated structures with disparate shapes and configurations. To this end, this paper proposes TransDeepLab, a novel DeepLab-like pure Transformer for medical image segmentation. Specifically, we exploit hierarchical Swin-Transformer with shifted windows to extend the DeepLabv3 and model the Atrous Spatial Pyramid Pooling (ASPP) module. A thorough search of the relevant literature yielded that we are the first to model the seminal DeepLab model with a pure Transformer-based model. Extensive experiments on various medical image segmentation tasks verify that our approach performs superior or on par with most contemporary works on an amalgamation of Vision Transformer and CNN-based methods, along with a significant reduction of model complexity. The codes and trained models are publicly available at https://github.com/rezazad68/transdeeplab