# HMT-UNet: A hybird Mamba-Transformer Vision UNet for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2408.11289 · HTML: https://mlsanity.com/paper/2408.11289/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Mamba for medical image analysis, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Mamba for medical image analysis](https://mlsanity.com/ssm/mamba-for-medical-image-analysis/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](https://mlsanity.com/paper/2407.08083/) (2024)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [Shallow Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2108.00882/) (2021)
- [MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation](https://mlsanity.com/paper/2211.01784/) (2022)
- [VM-UNET-V2 Rethinking Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2403.09157/) (2024)
- [ISIC 2017 - Skin Lesion Analysis Towards Melanoma Detection](https://mlsanity.com/paper/1703.00523/) (2017)
- [U-Net v2: Rethinking the Skip Connections of U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2311.17791/) (2023)

## Built on by (most cited first, 6 of 6)

- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [Image Augmentation Agent for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2412.20439/) (2024)
- [ASP-VMUNet: Atrous Shifted Parallel Vision Mamba U-Net for Skin Lesion Segmentation](https://mlsanity.com/paper/2503.19427/) (2025)
- [PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applications](https://mlsanity.com/paper/2512.01383/) (2025)
- [EAGLE: An Efficient Global Attention Lesion Segmentation Model for Hepatic Echinococcosis](https://mlsanity.com/paper/2506.20333/) (2025)
- [SpectralMamba-UNet: Frequency-Disentangled State Space Modeling for Texture-Structure Consistent Medical Image Segmentation](https://mlsanity.com/paper/2602.23103/) (2026)

## Abstract (arXiv)

In the field of medical image segmentation, models based on both CNN and Transformer have been thoroughly investigated. However, CNNs have limited modeling capabilities for long-range dependencies, making it challenging to exploit the semantic information within images fully. On the other hand, the quadratic computational complexity poses a challenge for Transformers. State Space Models (SSMs), such as Mamba, have been recognized as a promising method. They not only demonstrate superior performance in modeling long-range interactions, but also preserve a linear computational complexity. The hybrid mechanism of SSM (State Space Model) and Transformer, after meticulous design, can enhance its capability for efficient modeling of visual features. Extensive experiments have demonstrated that integrating the self-attention mechanism into the hybrid part behind the layers of Mamba's architecture can greatly improve the modeling capacity to capture long-range spatial dependencies. In this paper, leveraging the hybrid mechanism of SSM, we propose a U-shape architecture model for medical image segmentation, named Hybird Transformer vision Mamba UNet (HTM-UNet). We conduct comprehensive experiments on the ISIC17, ISIC18, CVC-300, CVC-ClinicDB, Kvasir, CVC-ColonDB, ETIS-Larib PolypDB public datasets and ZD-LCI-GIM private dataset. The results indicate that HTM-UNet exhibits competitive performance in medical image segmentation tasks. Our code is available at https://github.com/simzhangbest/HMT-Unet.