# Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentation

arXiv: https://arxiv.org/abs/2405.18511 · HTML: https://mlsanity.com/paper/2405.18511/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 14 of 14)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models](https://mlsanity.com/paper/2206.03461/) (2022)
- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation](https://mlsanity.com/paper/2303.14444/) (2023)
- [Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CT](https://mlsanity.com/paper/2009.02577/) (2020)
- [Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities](https://mlsanity.com/paper/1704.03379/) (2017)
- [Domain Composition and Attention for Unseen-Domain Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2109.08852/) (2021)
- [Glioblastoma Multiforme Prognosis: MRI Missing Modality Generation, Segmentation and Radiogenomic Survival Prediction](https://mlsanity.com/paper/2104.01149/) (2021)

## Built on by (most cited first, 8 of 8)

- [Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalities](https://mlsanity.com/paper/2406.11636/) (2024)
- [Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B](https://mlsanity.com/paper/2504.04918/) (2025)
- [Deep Unsupervised Anomaly Detection in Brain Imaging: Large-Scale Benchmarking and Bias Analysis](https://mlsanity.com/paper/2512.01534/) (2025)
- [Training-Free Zero-Shot Anomaly Detection in 3D Brain MRI with 2D Foundation Models](https://mlsanity.com/paper/2602.15315/) (2026)
- [Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training](https://mlsanity.com/paper/2509.09290/) (2025)
- [brat: Aligned Multi-View Embeddings for Brain MRI Analysis](https://mlsanity.com/paper/2512.18679/) (2025)
- [Toward Training-Free Zero-Shot Anomaly Detection in 3D Medical Images: A Batch-Based Approach Using 2D Foundation Models](https://mlsanity.com/paper/2606.18749/) (2026)
- [BrainIAC: Interactive 3D Brain Lesion Segmentation across Heterogeneous MRI Modalities with Online Adaptation](https://mlsanity.com/paper/2609.23026/) (2026)

## Abstract (arXiv)

Models for segmentation of brain lesions in multi-modal MRI are commonly trained for a specific pathology using a single database with a predefined set of MRI modalities, determined by a protocol for the specific disease. This work explores the following open questions: Is it feasible to train a model using multiple databases that contain varying sets of MRI modalities and annotations for different brain pathologies? Will this joint learning benefit performance on the sets of modalities and pathologies available during training? Will it enable analysis of new databases with different sets of modalities and pathologies? We develop and compare different methods and show that promising results can be achieved with appropriate, simple and practical alterations to the model and training framework. We experiment with 7 databases containing 5 types of brain pathologies and different sets of MRI modalities. Results demonstrate, for the first time, that joint training on multi-modal MRI databases with different brain pathologies and sets of modalities is feasible and offers practical benefits. It enables a single model to segment pathologies encountered during training in diverse sets of modalities, while facilitating segmentation of new types of pathologies such as via follow-up fine-tuning. The insights this study provides into the potential and limitations of this paradigm should prove useful for guiding future advances in the direction. Code and pretrained models: https://github.com/WenTXuL/MultiUnet