# A review: Deep learning for medical image segmentation using multi-modality fusion

arXiv: https://arxiv.org/abs/2004.10664 · HTML: https://mlsanity.com/paper/2004.10664/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 30 of 84)

- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2106.14591/) (2021)
- [M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization](https://mlsanity.com/paper/2307.08347/) (2023)
- [Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities](https://mlsanity.com/paper/2104.06231/) (2021)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications](https://mlsanity.com/paper/2408.02686/) (2024)
- [MyoPS-Net: Myocardial Pathology Segmentation with Flexible Combination of Multi-Sequence CMR Images](https://mlsanity.com/paper/2211.03062/) (2022)
- [Modality specific U-Net variants for biomedical image segmentation: A survey](https://mlsanity.com/paper/2107.04537/) (2021)
- [Multi-Modality Cardiac Image Computing: A Survey](https://mlsanity.com/paper/2208.12881/) (2022)
- [Noisy Labels are Treasure: Mean-Teacher-Assisted Confident Learning for Hepatic Vessel Segmentation](https://mlsanity.com/paper/2106.01860/) (2021)
- [Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities](https://mlsanity.com/paper/2111.04735/) (2021)
- [FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning](https://mlsanity.com/paper/2309.10283/) (2023)
- [Multimodal Information Fusion for Glaucoma and DR Classification](https://mlsanity.com/paper/2209.00979/) (2022)
- [Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images](https://mlsanity.com/paper/2308.07688/) (2023)
- [A Tri-attention Fusion Guided Multi-modal Segmentation Network](https://mlsanity.com/paper/2111.01623/) (2021)
- [COROLLA: An Efficient Multi-Modality Fusion Framework with Supervised Contrastive Learning for Glaucoma Grading](https://mlsanity.com/paper/2201.03795/) (2022)
- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [DoseDiff: Distance-aware Diffusion Model for Dose Prediction in Radiotherapy](https://mlsanity.com/paper/2306.16324/) (2023)
- [OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data](https://mlsanity.com/paper/2110.10640/) (2021)
- [Aligning Multi-Sequence CMR Towards Fully Automated Myocardial Pathology Segmentation](https://mlsanity.com/paper/2302.03537/) (2023)
- [Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation](https://mlsanity.com/paper/2307.08536/) (2023)
- [GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation](https://mlsanity.com/paper/2407.05540/) (2024)
- [MouseGAN++: Unsupervised Disentanglement and Contrastive Representation for Multiple MRI Modalities Synthesis and Structural Segmentation of Mouse Brain](https://mlsanity.com/paper/2212.01825/) (2022)
- [H-DenseFormer: An Efficient Hybrid Densely Connected Transformer for Multimodal Tumor Segmentation](https://mlsanity.com/paper/2307.01486/) (2023)
- [LMBiS-Net: A Lightweight Multipath Bidirectional Skip Connection based CNN for Retinal Blood Vessel Segmentation](https://mlsanity.com/paper/2309.04968/) (2023)
- [MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training](https://mlsanity.com/paper/2411.15576/) (2024)
- [Deep PET/CT fusion with Dempster-Shafer theory for lymphoma segmentation](https://mlsanity.com/paper/2108.05422/) (2021)
- [Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions](https://mlsanity.com/paper/2209.10307/) (2022)

## Abstract (arXiv)

Multi-modality is widely used in medical imaging, because it can provide multiinformation about a target (tumor, organ or tissue). Segmentation using multimodality consists of fusing multi-information to improve the segmentation. Recently, deep learning-based approaches have presented the state-of-the-art performance in image classification, segmentation, object detection and tracking tasks. Due to their self-learning and generalization ability over large amounts of data, deep learning recently has also gained great interest in multi-modal medical image segmentation. In this paper, we give an overview of deep learning-based approaches for multi-modal medical image segmentation task. Firstly, we introduce the general principle of deep learning and multi-modal medical image segmentation. Secondly, we present different deep learning network architectures, then analyze their fusion strategies and compare their results. The earlier fusion is commonly used, since it's simple and it focuses on the subsequent segmentation network architecture. However, the later fusion gives more attention on fusion strategy to learn the complex relationship between different modalities. In general, compared to the earlier fusion, the later fusion can give more accurate result if the fusion method is effective enough. We also discuss some common problems in medical image segmentation. Finally, we summarize and provide some perspectives on the future research.