# Multi-Compound Transformer for Accurate Biomedical Image Segmentation

arXiv: https://arxiv.org/abs/2106.14385 · HTML: https://mlsanity.com/paper/2106.14385/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [CE-Net: Context Encoder Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/1903.02740/) (2019)
- [Context Prior for Scene Segmentation](https://mlsanity.com/paper/2004.01547/) (2020)
- [Segmenting Transparent Object in the Wild with Transformer](https://mlsanity.com/paper/2101.08461/) (2021)
- [UXNet: Searching Multi-level Feature Aggregation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2009.07501/) (2020)

## Built on by (most cited first, 30 of 42)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer](https://mlsanity.com/paper/2109.04335/) (2021)
- [TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2107.05274/) (2021)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [ACC-UNet: A Completely Convolutional UNet model for the 2020s](https://mlsanity.com/paper/2308.13680/) (2023)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [TransFusion: Multi-view Divergent Fusion for Medical Image Segmentation with Transformers](https://mlsanity.com/paper/2203.10726/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [LaB-GATr: geometric algebra transformers for large biomedical surface and volume meshes](https://mlsanity.com/paper/2403.07536/) (2024)
- [Do we really need temporal convolutions in action segmentation?](https://mlsanity.com/paper/2205.13425/) (2022)
- [The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning](https://mlsanity.com/paper/2206.14413/) (2022)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [Narrowing the semantic gaps in U-Net with learnable skip connections: The case of medical image segmentation](https://mlsanity.com/paper/2312.15182/) (2023)
- [MaxViT-UNet: Multi-Axis Attention for Medical Image Segmentation](https://mlsanity.com/paper/2305.08396/) (2023)
- [The Multiscale Surface Vision Transformer](https://mlsanity.com/paper/2303.11909/) (2023)
- [Revisiting Computer-Aided Tuberculosis Diagnosis](https://mlsanity.com/paper/2307.02848/) (2023)
- [Toward Clinically Assisted Colorectal Polyp Recognition via Structured Cross-modal Representation Consistency](https://mlsanity.com/paper/2206.11826/) (2022)
- [MCANet: Medical Image Segmentation with Multi-Scale Cross-Axis Attention](https://mlsanity.com/paper/2312.08866/) (2023)
- [HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2412.11458/) (2024)
- [ConvTransSeg: A Multi-resolution Convolution-Transformer Network for Medical Image Segmentation](https://mlsanity.com/paper/2210.07072/) (2022)
- [TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction](https://mlsanity.com/paper/2301.10829/) (2023)
- [HST-MRF: Heterogeneous Swin Transformer with Multi-Receptive Field for Medical Image Segmentation](https://mlsanity.com/paper/2304.04614/) (2023)
- [Crop and Couple: cardiac image segmentation using interlinked specialist networks](https://mlsanity.com/paper/2402.09156/) (2024)
- [SkinFormer: Learning Statistical Texture Representation with Transformer for Skin Lesion Segmentation](https://mlsanity.com/paper/2409.08652/) (2024)
- [Debunking Grad-ECLIP: A Comprehensive Study on Its Incorrectness and Fundamental Principles for Model Interpretation](https://mlsanity.com/paper/2605.12952/) (2026)
- [Memory-efficient Segmentation of High-resolution Volumetric MicroCT Images](https://mlsanity.com/paper/2205.15941/) (2022)
- [A New Perspective to Boost Vision Transformer for Medical Image Classification](https://mlsanity.com/paper/2301.00989/) (2023)
- [LCAUnet: A skin lesion segmentation network with enhanced edge and body fusion](https://mlsanity.com/paper/2305.00837/) (2023)
- [Q2A: Querying Implicit Fully Continuous Feature Pyramid to Align Features for Medical Image Segmentation](https://mlsanity.com/paper/2404.09472/) (2024)

## Abstract (arXiv)

The recent vision transformer(i.e.for image classification) learns non-local attentive interaction of different patch tokens. However, prior arts miss learning the cross-scale dependencies of different pixels, the semantic correspondence of different labels, and the consistency of the feature representations and semantic embeddings, which are critical for biomedical segmentation. In this paper, we tackle the above issues by proposing a unified transformer network, termed Multi-Compound Transformer (MCTrans), which incorporates rich feature learning and semantic structure mining into a unified framework. Specifically, MCTrans embeds the multi-scale convolutional features as a sequence of tokens and performs intra- and inter-scale self-attention, rather than single-scale attention in previous works. In addition, a learnable proxy embedding is also introduced to model semantic relationship and feature enhancement by using self-attention and cross-attention, respectively. MCTrans can be easily plugged into a UNet-like network and attains a significant improvement over the state-of-the-art methods in biomedical image segmentation in six standard benchmarks. For example, MCTrans outperforms UNet by 3.64%, 3.71%, 4.34%, 2.8%, 1.88%, 1.57% in Pannuke, CVC-Clinic, CVC-Colon, Etis, Kavirs, ISIC2018 dataset, respectively. Code is available at https://github.com/JiYuanFeng/MCTrans.