# SegFormer3D: an Efficient Transformer for 3D Medical Image Segmentation

arXiv: https://arxiv.org/abs/2404.10156 · HTML: https://mlsanity.com/paper/2404.10156/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2107.08623/) (2021)
- [3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes](https://mlsanity.com/paper/1607.00582/) (2016)
- [TransClaw U-Net: Claw U-Net with Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2107.05188/) (2021)
- [Deeply-Supervised CNN for Prostate Segmentation](https://mlsanity.com/paper/1703.07523/) (2017)
- [Hierarchical 3D fully convolutional networks for multi-organ segmentation](https://mlsanity.com/paper/1704.06382/) (2017)
- [Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain](https://mlsanity.com/paper/2402.06190/) (2024)
- [PGD-UNet: A Position-Guided Deformable Network for Simultaneous Segmentation of Organs and Tumors](https://mlsanity.com/paper/2007.01001/) (2020)

## Built on by (most cited first, 30 of 31)

- [DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks](https://mlsanity.com/paper/2409.06809/) (2024)
- [Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning](https://mlsanity.com/paper/2509.19090/) (2025)
- [KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models](https://mlsanity.com/paper/2410.01595/) (2024)
- [MobileUNETR: A Lightweight End-To-End Hybrid Vision Transformer For Efficient Medical Image Segmentation](https://mlsanity.com/paper/2409.03062/) (2024)
- [No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-k Patch Sampling](https://mlsanity.com/paper/2501.10814/) (2025)
- [Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning](https://mlsanity.com/paper/2409.10362/) (2024)
- [A Comprehensive Analysis of Mamba for 3D Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2503.19308/) (2025)
- [Federated nnU-Net for Privacy-Preserving Medical Image Segmentation](https://mlsanity.com/paper/2503.02549/) (2025)
- [FuseUNet: A Multi-Scale Feature Fusion Method for U-like Networks](https://mlsanity.com/paper/2506.05821/) (2025)
- [DocParseNet: Advanced Semantic Segmentation and OCR Embeddings for Efficient Scanned Document Annotation](https://mlsanity.com/paper/2406.17591/) (2024)
- [Interpretable Auto Window Setting for Deep-Learning-Based CT Analysis](https://mlsanity.com/paper/2501.06223/) (2025)
- [Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI](https://mlsanity.com/paper/2604.15808/) (2026)
- [Higher-Order Transformers With Kronecker-Structured Attention](https://mlsanity.com/paper/2412.02919/) (2024)
- [OmniMamba4D: Spatio-temporal Mamba for longitudinal CT lesion segmentation](https://mlsanity.com/paper/2504.09655/) (2025)
- [SegFormer Fine-Tuning with Dropout: Advancing Hair Artifact Removal in Skin Lesion Analysis](https://mlsanity.com/paper/2509.02156/) (2025)
- [Spatial-Aware Self-Supervision for Medical 3D Imaging with Multi-Granularity Observable Tasks](https://mlsanity.com/paper/2509.05967/) (2025)
- [Frequency-Aware Ensemble Learning for BraTS 2025 Pediatric Brain Tumor Segmentation](https://mlsanity.com/paper/2509.19353/) (2025)
- [SEMIR: Semantic Minor-Induced Representation Learning on Graphs for Visual Segmentation](https://mlsanity.com/paper/2605.12389/) (2026)
- [IntelliCardiac: An Intelligent Platform for Cardiac Image Segmentation and Classification](https://mlsanity.com/paper/2505.03838/) (2025)
- [GaMNet: A Hybrid Network with Gabor Fusion and NMamba for Efficient 3D Glioma Segmentation](https://mlsanity.com/paper/2505.05520/) (2025)
- [DM-SegNet: Dual-Mamba Architecture for 3D Medical Image Segmentation with Global Context Modeling](https://mlsanity.com/paper/2506.05297/) (2025)
- [Graph-Based Multi-Modal Light-weight Network for Adaptive Brain Tumor Segmentation](https://mlsanity.com/paper/2507.09995/) (2025)
- [ASM-UNet: Adaptive Scan Mamba Integrating Group Commonalities and Individual Variations for Fine-Grained Segmentation](https://mlsanity.com/paper/2508.07237/) (2025)
- [Johnson-Lindenstrauss Lemma Guided Network for Efficient 3D Medical Segmentation](https://mlsanity.com/paper/2509.22307/) (2025)
- [PF-DAformer: Proximal Femur Segmentation via Domain Adaptive Transformer for Dual-Center QCT](https://mlsanity.com/paper/2510.26903/) (2025)
- [Breast Cancer Neoadjuvant Chemotherapy Treatment Response Prediction Using Aligned Longitudinal MRI and Clinical Data](https://mlsanity.com/paper/2512.17759/) (2025)
- [RefineFormer3D: Efficient 3D Medical Image Segmentation via Adaptive Multi-Scale Transformer with Cross Attention Fusion](https://mlsanity.com/paper/2602.16320/) (2026)
- [ILV: Iterative Latent Volumes for Fast and Accurate Sparse-View CT Reconstruction](https://mlsanity.com/paper/2603.14915/) (2026)
- [SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation](https://mlsanity.com/paper/2603.22002/) (2026)
- [GPAFormer: Graph-guided Patch Aggregation Transformer for Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2604.06658/) (2026)

## Abstract (arXiv)

The adoption of Vision Transformers (ViTs) based architectures represents a significant advancement in 3D Medical Image (MI) segmentation, surpassing traditional Convolutional Neural Network (CNN) models by enhancing global contextual understanding. While this paradigm shift has significantly enhanced 3D segmentation performance, state-of-the-art architectures require extremely large and complex architectures with large scale computing resources for training and deployment. Furthermore, in the context of limited datasets, often encountered in medical imaging, larger models can present hurdles in both model generalization and convergence. In response to these challenges and to demonstrate that lightweight models are a valuable area of research in 3D medical imaging, we present SegFormer3D, a hierarchical Transformer that calculates attention across multiscale volumetric features. Additionally, SegFormer3D avoids complex decoders and uses an all-MLP decoder to aggregate local and global attention features to produce highly accurate segmentation masks. The proposed memory efficient Transformer preserves the performance characteristics of a significantly larger model in a compact design. SegFormer3D democratizes deep learning for 3D medical image segmentation by offering a model with 33x less parameters and a 13x reduction in GFLOPS compared to the current state-of-the-art (SOTA). We benchmark SegFormer3D against the current SOTA models on three widely used datasets Synapse, BRaTs, and ACDC, achieving competitive results. Code: https://github.com/OSUPCVLab/SegFormer3D.git