# 3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

arXiv: https://arxiv.org/abs/2304.14508 · HTML: https://mlsanity.com/paper/2304.14508/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation](https://mlsanity.com/paper/1912.05074/) (2019)
- [UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation](https://mlsanity.com/paper/2004.08790/) (2020)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation](https://mlsanity.com/paper/2107.00781/) (2021)
- [Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks](https://mlsanity.com/paper/1808.08127/) (2018)
- [Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation](https://mlsanity.com/paper/1711.01468/) (2017)
- [Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge](https://mlsanity.com/paper/1802.10508/) (2018)
- [On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task](https://mlsanity.com/paper/1707.01992/) (2017)
- [Adaptive feature recombination and recalibration for semantic segmentation with Fully Convolutional Networks](https://mlsanity.com/paper/2006.11193/) (2020)
- [MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet](https://mlsanity.com/paper/2206.00902/) (2022)
- [Global Planar Convolutions for improved context aggregation in Brain Tumor Segmentation](https://mlsanity.com/paper/1908.10281/) (2019)
- [Survival prediction using ensemble tumor segmentation and transfer learning](https://mlsanity.com/paper/1810.04274/) (2018)

## Built on by (most cited first, 1 of 1)

- [Vision Transformers increase efficiency of 3D cardiac CT multi-label segmentation](https://mlsanity.com/paper/2310.09099/) (2023)

## Abstract (arXiv)

Magnetic resonance imaging (MRI) is critically important for brain mapping in both scientific research and clinical studies. Precise segmentation of brain tumors facilitates clinical diagnosis, evaluations, and surgical planning. Deep learning has recently emerged to improve brain tumor segmentation and achieved impressive results. Convolutional architectures are widely used to implement those neural networks. By the nature of limited receptive fields, however, those architectures are subject to representing long-range spatial dependencies of the voxel intensities in MRI images. Transformers have been leveraged recently to address the above limitations of convolutional networks. Unfortunately, the majority of current Transformers-based methods in segmentation are performed with 2D MRI slices, instead of 3D volumes. Moreover, it is difficult to incorporate the structures between layers because each head is calculated independently in the Multi-Head Self-Attention mechanism (MHSA). In this work, we proposed a 3D Transformer-based segmentation approach. We developed a Fusion-Head Self-Attention mechanism (FHSA) to combine each attention head through attention logic and weight mapping, for the exploration of the long-range spatial dependencies in 3D MRI images. We implemented a plug-and-play self-attention module, named the Infinite Deformable Fusion Transformer Module (IDFTM), to extract features on any deformable feature maps. We applied our approach to the task of brain tumor segmentation, and assessed it on the public BRATS datasets. The experimental results demonstrated that our proposed approach achieved superior performance, in comparison to several state-of-the-art segmentation methods.