# Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining

arXiv: https://arxiv.org/abs/2402.03302 · HTML: https://mlsanity.com/paper/2402.03302/

## Key facts

- Cited by 179 papers in the map (data through 14 Oct 2026).
- Placed in Mamba for medical image analysis, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Mamba for medical image analysis](https://mlsanity.com/ssm/mamba-for-medical-image-analysis/)

## Builds on (most cited first, 30 of 33)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](https://mlsanity.com/paper/2109.03201/) (2021)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Global Context Vision Transformers](https://mlsanity.com/paper/2206.09959/) (2022)
- [Multi-scale self-guided attention for medical image segmentation](https://mlsanity.com/paper/1906.02849/) (2019)
- [Annotation-efficient deep learning for automatic medical image segmentation](https://mlsanity.com/paper/2012.04885/) (2020)
- [Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge](https://mlsanity.com/paper/2308.05862/) (2023)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions](https://mlsanity.com/paper/2308.05864/) (2023)
- [Fully Convolutional Multi-scale Residual DenseNets for Cardiac Segmentation and Automated Cardiac Diagnosis using Ensemble of Classifiers](https://mlsanity.com/paper/1801.05173/) (2018)
- [D-UNet: a dimension-fusion U shape network for chronic stroke lesion segmentation](https://mlsanity.com/paper/1908.05104/) (2019)
- [X-Net: Brain Stroke Lesion Segmentation Based on Depthwise Separable Convolution and Long-range Dependencies](https://mlsanity.com/paper/1907.07000/) (2019)
- [AUNet: Attention-guided dense-upsampling networks for breast mass segmentation in whole mammograms](https://mlsanity.com/paper/1810.10151/) (2018)

## Built on by (most cited first, 30 of 179)

- [LocalMamba: Visual State Space Model with Windowed Selective Scan](https://mlsanity.com/paper/2403.09338/) (2024)
- [PointMamba: A Simple State Space Model for Point Cloud Analysis](https://mlsanity.com/paper/2402.10739/) (2024)
- [EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba](https://mlsanity.com/paper/2403.09977/) (2024)
- [PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition](https://mlsanity.com/paper/2403.17695/) (2024)
- [MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2404.06564/) (2024)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [Graph Mamba: Towards Learning on Graphs with State Space Models](https://mlsanity.com/paper/2402.08678/) (2024)
- [ZigMa: A DiT-style Zigzag Mamba Diffusion Model](https://mlsanity.com/paper/2403.13802/) (2024)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection](https://mlsanity.com/paper/2403.02148/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Vision Mamba: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2405.04404/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [Mamba-R: Vision Mamba ALSO Needs Registers](https://mlsanity.com/paper/2405.14858/) (2024)
- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [ReMamber: Referring Image Segmentation with Mamba Twister](https://mlsanity.com/paper/2403.17839/) (2024)
- [SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model](https://mlsanity.com/paper/2405.11831/) (2024)
- [VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting](https://mlsanity.com/paper/2403.16536/) (2024)
- [RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation](https://mlsanity.com/paper/2404.02457/) (2024)
- [MambaLRP: Explaining Selective State Space Sequence Models](https://mlsanity.com/paper/2406.07592/) (2024)
- [SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation](https://mlsanity.com/paper/2412.11890/) (2024)
- [I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling](https://mlsanity.com/paper/2405.14022/) (2024)
- [Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation](https://mlsanity.com/paper/2601.10880/) (2026)
- [STG-Mamba: Spatial-Temporal Graph Learning via Selective State Space Model](https://mlsanity.com/paper/2403.12418/) (2024)
- [Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Models](https://mlsanity.com/paper/2409.07163/) (2024)
- [Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning](https://mlsanity.com/paper/2402.15761/) (2024)
- [Autoregressive Pretraining with Mamba in Vision](https://mlsanity.com/paper/2406.07537/) (2024)
- [DGMamba: Domain Generalization via Generalized State Space Model](https://mlsanity.com/paper/2404.07794/) (2024)
- [PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learning](https://mlsanity.com/paper/2405.15214/) (2024)

## Abstract (arXiv)

Accurate medical image segmentation demands the integration of multi-scale information, spanning from local features to global dependencies. However, it is challenging for existing methods to model long-range global information, where convolutional neural networks (CNNs) are constrained by their local receptive fields, and vision transformers (ViTs) suffer from high quadratic complexity of their attention mechanism. Recently, Mamba-based models have gained great attention for their impressive ability in long sequence modeling. Several studies have demonstrated that these models can outperform popular vision models in various tasks, offering higher accuracy, lower memory consumption, and less computational burden. However, existing Mamba-based models are mostly trained from scratch and do not explore the power of pretraining, which has been proven to be quite effective for data-efficient medical image analysis. This paper introduces a novel Mamba-based model, Swin-UMamba, designed specifically for medical image segmentation tasks, leveraging the advantages of ImageNet-based pretraining. Our experimental results reveal the vital role of ImageNet-based training in enhancing the performance of Mamba-based models. Swin-UMamba demonstrates superior performance with a large margin compared to CNNs, ViTs, and latest Mamba-based models. Notably, on AbdomenMRI, Encoscopy, and Microscopy datasets, Swin-UMamba outperforms its closest counterpart U-Mamba_Enc by an average score of 2.72%.