# A 3D generative model of pathological multi-modal MR images and segmentations

arXiv: https://arxiv.org/abs/2311.04552 · HTML: https://mlsanity.com/paper/2311.04552/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 12 of 12)

- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Robust machine learning segmentation for large-scale analysis of heterogeneous clinical brain MRI datasets](https://mlsanity.com/paper/2209.02032/) (2022)
- [Can segmentation models be trained with fully synthetically generated data?](https://mlsanity.com/paper/2209.08256/) (2022)
- [Red-GAN: Attacking class imbalance via conditioned generation. Yet another perspective on medical image synthesis for skin lesion dermoscopy and brain tumor MRI](https://mlsanity.com/paper/2004.10734/) (2020)
- [Morphology-preserving Autoregressive 3D Generative Modelling of the Brain](https://mlsanity.com/paper/2209.03177/) (2022)
- [Synthesizing brain tumor images and annotations by combining progressive growing GAN and SPADE](https://mlsanity.com/paper/2009.05946/) (2020)

## Built on by (most cited first, 4 of 4)

- [Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models](https://mlsanity.com/paper/2402.05210/) (2024)
- [Anatomical feature-prioritized loss for enhanced MR to CT translation](https://mlsanity.com/paper/2410.10328/) (2024)
- [Diffusion Models for conditional MRI generation](https://mlsanity.com/paper/2502.18620/) (2025)
- [Guided Synthesis of Labeled Brain MRI Data Using Latent Diffusion Models for Segmentation of Enlarged Ventricles](https://mlsanity.com/paper/2411.01351/) (2024)

## Abstract (arXiv)

Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.