# Mixed Transformer U-Net For Medical Image Segmentation

arXiv: https://arxiv.org/abs/2111.04734 · HTML: https://mlsanity.com/paper/2111.04734/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation](https://mlsanity.com/paper/2004.08790/) (2020)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks](https://mlsanity.com/paper/2105.02358/) (2021)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [Domain Adaptive Relational Reasoning for 3D Multi-Organ Segmentation](https://mlsanity.com/paper/2005.09120/) (2020)
- [TransClaw U-Net: Claw U-Net with Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2107.05188/) (2021)

## Built on by (most cited first, 30 of 95)

- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2405.06880/) (2024)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2303.16892/) (2023)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [The Fully Convolutional Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2206.00566/) (2022)
- [Ladder Fine-tuning approach for SAM integrating complementary network](https://mlsanity.com/paper/2306.12737/) (2023)
- [Transformer Meets Boundary Value Inverse Problems](https://mlsanity.com/paper/2209.14977/) (2022)
- [G-CASCADE: Efficient Cascaded Graph Convolutional Decoding for 2D Medical Image Segmentation](https://mlsanity.com/paper/2310.16175/) (2023)
- [DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2310.12570/) (2023)
- [BATFormer: Towards Boundary-Aware Lightweight Transformer for Efficient Medical Image Segmentation](https://mlsanity.com/paper/2206.14409/) (2022)
- [Optimizing Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2210.08066/) (2022)
- [MIST: Medical Image Segmentation Transformer with Convolutional Attention Mixing (CAM) Decoder](https://mlsanity.com/paper/2310.19898/) (2023)
- [CSWin-UNet: Transformer UNet with Cross-Shaped Windows for Medical Image Segmentation](https://mlsanity.com/paper/2407.18070/) (2024)
- [MetaSeg: MetaFormer-based Global Contexts-aware Network for Efficient Semantic Segmentation](https://mlsanity.com/paper/2408.07576/) (2024)
- [SliceMamba with Neural Architecture Search for Medical Image Segmentation](https://mlsanity.com/paper/2407.08481/) (2024)
- [The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning](https://mlsanity.com/paper/2206.14413/) (2022)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [D-Net: Dynamic Large Kernel with Dynamic Feature Fusion for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2403.10674/) (2024)
- [Vision Foundation Models in Medical Image Analysis: Advances and Challenges](https://mlsanity.com/paper/2502.14584/) (2025)
- [MEW-UNet: Multi-axis representation learning in frequency domain for medical image segmentation](https://mlsanity.com/paper/2210.14007/) (2022)
- [PSFHS Challenge Report: Pubic Symphysis and Fetal Head Segmentation from Intrapartum Ultrasound Images](https://mlsanity.com/paper/2409.10980/) (2024)
- [BirdSoundsDenoising: Deep Visual Audio Denoising for Bird Sounds](https://mlsanity.com/paper/2210.10196/) (2022)
- [RWKV-UNet: Improving UNet with Long-Range Cooperation for Effective Medical Image Segmentation](https://mlsanity.com/paper/2501.08458/) (2025)
- [UNet-2022: Exploring Dynamics in Non-isomorphic Architecture](https://mlsanity.com/paper/2210.15566/) (2022)
- [A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysis](https://mlsanity.com/paper/2406.06136/) (2024)
- [CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation](https://mlsanity.com/paper/2408.13698/) (2024)
- [LV-UNet: A Lightweight and Vanilla Model for Medical Image Segmentation](https://mlsanity.com/paper/2408.16886/) (2024)

## Abstract (arXiv)

Though U-Net has achieved tremendous success in medical image segmentation tasks, it lacks the ability to explicitly model long-range dependencies. Therefore, Vision Transformers have emerged as alternative segmentation structures recently, for their innate ability of capturing long-range correlations through Self-Attention (SA). However, Transformers usually rely on large-scale pre-training and have high computational complexity. Furthermore, SA can only model self-affinities within a single sample, ignoring the potential correlations of the overall dataset. To address these problems, we propose a novel Transformer module named Mixed Transformer Module (MTM) for simultaneous inter- and intra- affinities learning. MTM first calculates self-affinities efficiently through our well-designed Local-Global Gaussian-Weighted Self-Attention (LGG-SA). Then, it mines inter-connections between data samples through External Attention (EA). By using MTM, we construct a U-shaped model named Mixed Transformer U-Net (MT-UNet) for accurate medical image segmentation. We test our method on two different public datasets, and the experimental results show that the proposed method achieves better performance over other state-of-the-art methods. The code is available at: https://github.com/Dootmaan/MT-UNet.