# PHTrans: Parallelly Aggregating Global and Local Representations for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2203.04568 · HTML: https://mlsanity.com/paper/2203.04568/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network](https://mlsanity.com/paper/1703.02719/) (2017)
- [MISSFormer: An Effective Medical Image Segmentation Transformer](https://mlsanity.com/paper/2109.07162/) (2021)
- [LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2107.08623/) (2021)

## Built on by (most cited first, 16 of 16)

- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [Automatic Organ and Pan-cancer Segmentation in Abdomen CT: the FLARE 2023 Challenge](https://mlsanity.com/paper/2408.12534/) (2024)
- [NexToU: Efficient Topology-Aware U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2305.15911/) (2023)
- [DIAS: A Dataset and Benchmark for Intracranial Artery Segmentation in DSA sequences](https://mlsanity.com/paper/2306.12153/) (2023)
- [Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer](https://mlsanity.com/paper/2307.03427/) (2023)
- [ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation](https://mlsanity.com/paper/2309.05674/) (2023)
- [UNet-2022: Exploring Dynamics in Non-isomorphic Architecture](https://mlsanity.com/paper/2210.15566/) (2022)
- [HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2412.11458/) (2024)
- [Combining Self-Training and Hybrid Architecture for Semi-supervised Abdominal Organ Segmentation](https://mlsanity.com/paper/2207.11512/) (2022)
- [Two-Stage Hybrid Supervision Framework for Fast, Low-resource, and Accurate Organ and Pan-cancer Segmentation in Abdomen CT](https://mlsanity.com/paper/2309.05405/) (2023)
- [UDCR: Unsupervised Aortic DSA/CTA Rigid Registration Using Deep Reinforcement Learning and Overlap Degree Calculation](https://mlsanity.com/paper/2403.05753/) (2024)
- [GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation](https://mlsanity.com/paper/2404.17854/) (2024)
- [Learning to In-paint: Domain Adaptive Shape Completion for 3D Organ Segmentation](https://mlsanity.com/paper/2308.08775/) (2023)
- [MetaFormer-driven Encoding Network for Robust Medical Semantic Segmentation](https://mlsanity.com/paper/2601.00922/) (2026)
- [Semantic Class Distribution Learning for Debiasing Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2603.05202/) (2026)
- [SwInception -- Local Attention Meets Convolutions](https://mlsanity.com/paper/2605.29954/) (2026)

## Abstract (arXiv)

The success of Transformer in computer vision has attracted increasing attention in the medical imaging community. Especially for medical image segmentation, many excellent hybrid architectures based on convolutional neural networks (CNNs) and Transformer have been presented and achieve impressive performance. However, most of these methods, which embed modular Transformer into CNNs, struggle to reach their full potential. In this paper, we propose a novel hybrid architecture for medical image segmentation called PHTrans, which parallelly hybridizes Transformer and CNN in main building blocks to produce hierarchical representations from global and local features and adaptively aggregate them, aiming to fully exploit their strengths to obtain better segmentation performance. Specifically, PHTrans follows the U-shaped encoder-decoder design and introduces the parallel hybird module in deep stages, where convolution blocks and the modified 3D Swin Transformer learn local features and global dependencies separately, then a sequence-to-volume operation unifies the dimensions of the outputs to achieve feature aggregation. Extensive experimental results on both Multi-Atlas Labeling Beyond the Cranial Vault and Automated Cardiac Diagnosis Challeng datasets corroborate its effectiveness, consistently outperforming state-of-the-art methods. The code is available at: https://github.com/lseventeen/PHTrans.