# Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2303.16892 · HTML: https://mlsanity.com/paper/2303.16892/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 26 of 26)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation](https://mlsanity.com/paper/2004.08790/) (2020)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2106.06716/) (2021)
- [Reverse Attention for Salient Object Detection](https://mlsanity.com/paper/1807.09940/) (2018)
- [MISSFormer: An Effective Medical Image Segmentation Transformer](https://mlsanity.com/paper/2109.07162/) (2021)
- [Mixed Transformer U-Net For Medical Image Segmentation](https://mlsanity.com/paper/2111.04734/) (2021)
- [Stepwise Feature Fusion: Local Guides Global](https://mlsanity.com/paper/2203.03635/) (2022)
- [Class-Aware Adversarial Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2201.10737/) (2022)
- [DC-UNet: Rethinking the U-Net Architecture with Dual Channel Efficient CNN for Medical Images Segmentation](https://mlsanity.com/paper/2006.00414/) (2020)

## Built on by (most cited first, 30 of 34)

- [EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2405.06880/) (2024)
- [Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding](https://mlsanity.com/paper/2403.18271/) (2024)
- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation](https://mlsanity.com/paper/2404.02845/) (2024)
- [G-CASCADE: Efficient Cascaded Graph Convolutional Decoding for 2D Medical Image Segmentation](https://mlsanity.com/paper/2310.16175/) (2023)
- [HiDiff: Hybrid Diffusion Framework for Medical Image Segmentation](https://mlsanity.com/paper/2407.03548/) (2024)
- [MIST: Medical Image Segmentation Transformer with Convolutional Attention Mixing (CAM) Decoder](https://mlsanity.com/paper/2310.19898/) (2023)
- [MK-UNet: Multi-kernel Lightweight CNN for Medical Image Segmentation](https://mlsanity.com/paper/2509.18493/) (2025)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [Autoregressive Medical Image Segmentation via Next-Scale Mask Prediction](https://mlsanity.com/paper/2502.20784/) (2025)
- [PraNet-V2: Dual-Supervised Reverse Attention for Medical Image Segmentation](https://mlsanity.com/paper/2504.10986/) (2025)
- [Medical Image Segmentation via Sparse Coding Decoder](https://mlsanity.com/paper/2310.10957/) (2023)
- [AgileFormer: Spatially Agile Transformer UNet for Medical Image Segmentation](https://mlsanity.com/paper/2404.00122/) (2024)
- [PP-SAM: Perturbed Prompts for Robust Adaptation of Segment Anything Model for Polyp Segmentation](https://mlsanity.com/paper/2405.16740/) (2024)
- [AI-SAM: Automatic and Interactive Segment Anything Model](https://mlsanity.com/paper/2312.03119/) (2023)
- [PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation](https://mlsanity.com/paper/2503.18227/) (2025)
- [Pixel-wise Modulated Dice Loss for Medical Image Segmentation](https://mlsanity.com/paper/2506.15744/) (2025)
- [Multi-dimension Transformer with Attention-based Filtering for Medical Image Segmentation](https://mlsanity.com/paper/2405.12328/) (2024)
- [SACNet: A Spatially Adaptive Convolution Network for 2D Multi-organ Medical Segmentation](https://mlsanity.com/paper/2407.10157/) (2024)
- [Continuous and complete liver vessel segmentation with graph-attention guided diffusion](https://mlsanity.com/paper/2411.00617/) (2024)
- [Advanced Brain Tumor Segmentation Using EMCAD: Efficient Multi-scale Convolutional Attention Decoding](https://mlsanity.com/paper/2509.05431/) (2025)
- [Dual Interaction Network with Cross-Image Attention for Medical Image Segmentation](https://mlsanity.com/paper/2509.05953/) (2025)
- [LoMix: Learnable Weighted Multi-Scale Logits Mixing for Medical Image Segmentation](https://mlsanity.com/paper/2510.22995/) (2025)
- [An Effective UNet Using Feature Interaction and Fusion for Organ Segmentation in Medical Image](https://mlsanity.com/paper/2409.05324/) (2024)
- [Advancing Oyster Phenotype Segmentation with Multi-Network Ensemble and Multi-Scale mechanism](https://mlsanity.com/paper/2501.11203/) (2025)
- [Uncertainty-Based Ensemble Learning in CMR Semantic Segmentation](https://mlsanity.com/paper/2502.09269/) (2025)
- [MambaCAFU: Hybrid Multi-Scale and Multi-Attention Model with Mamba-Based Fusion for Medical Image Segmentation](https://mlsanity.com/paper/2510.03786/) (2025)
- [SA2Net: Scale-Adaptive Structure-Affinity Transformation for Spine Segmentation from Ultrasound Volume Projection Imaging](https://mlsanity.com/paper/2510.26568/) (2025)
- [SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images](https://mlsanity.com/paper/2511.08626/) (2025)
- [Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation](https://mlsanity.com/paper/2603.02727/) (2026)

## Abstract (arXiv)

Transformers have shown great success in medical image segmentation. However, transformers may exhibit a limited generalization ability due to the underlying single-scale self-attention (SA) mechanism. In this paper, we address this issue by introducing a Multi-scale hiERarchical vIsion Transformer (MERIT) backbone network, which improves the generalizability of the model by computing SA at multiple scales. We also incorporate an attention-based decoder, namely Cascaded Attention Decoding (CASCADE), for further refinement of multi-stage features generated by MERIT. Finally, we introduce an effective multi-stage feature mixing loss aggregation (MUTATION) method for better model training via implicit ensembling. Our experiments on two widely used medical image segmentation benchmarks (i.e., Synapse Multi-organ, ACDC) demonstrate the superior performance of MERIT over state-of-the-art methods. Our MERIT architecture and MUTATION loss aggregation can be used with downstream medical image and semantic segmentation tasks.