# Hybrid-Fusion Transformer for Multisequence MRI

arXiv: https://arxiv.org/abs/2311.01308 · HTML: https://mlsanity.com/paper/2311.01308/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [nnU-Net for Brain Tumor Segmentation](https://mlsanity.com/paper/2011.00848/) (2020)
- [SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection](https://mlsanity.com/paper/2204.05585/) (2022)
- [CNN-based Segmentation of Medical Imaging Data](https://mlsanity.com/paper/1701.03056/) (2017)
- [TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network](https://mlsanity.com/paper/2108.03990/) (2021)

## Built on by (most cited first, 6 of 6)

- [Two-Stage Approach for Brain MR Image Synthesis: 2D Image Synthesis and 3D Refinement](https://mlsanity.com/paper/2410.10269/) (2024)
- [Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser](https://mlsanity.com/paper/2402.00375/) (2024)
- [Label-Efficient 3D Brain Segmentation via Complementary 2D Diffusion Models with Orthogonal Views](https://mlsanity.com/paper/2407.12329/) (2024)
- [Principled Feature Disentanglement for High-Fidelity Unified Brain MRI Synthesis](https://mlsanity.com/paper/2406.14954/) (2024)
- [Selective Complementary Feature Fusion and Modal Feature Compression Interaction for Brain Tumor Segmentation](https://mlsanity.com/paper/2503.16149/) (2025)
- [Learning What is Worth Learning: Active and Sequential Domain Adaptation for Multi-modal Gross Tumor Volume Segmentation](https://mlsanity.com/paper/2508.20528/) (2025)

## Abstract (arXiv)

Medical segmentation has grown exponentially through the advent of a fully convolutional network (FCN), and we have now reached a turning point through the success of Transformer. However, the different characteristics of the modality have not been fully integrated into Transformer for medical segmentation. In this work, we propose the novel hybrid fusion Transformer (HFTrans) for multisequence MRI image segmentation. We take advantage of the differences among multimodal MRI sequences and utilize the Transformer layers to integrate the features extracted from each modality as well as the features of the early fused modalities. We validate the effectiveness of our hybrid-fusion method in three-dimensional (3D) medical segmentation. Experiments on two public datasets, BraTS2020 and MRBrainS18, show that the proposed method outperforms previous state-of-the-art methods on the task of brain tumor segmentation and brain structure segmentation.