# Can segmentation models be trained with fully synthetically generated data?

arXiv: https://arxiv.org/abs/2209.08256 · HTML: https://mlsanity.com/paper/2209.08256/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [StyleBank: An Explicit Representation for Neural Image Style Transfer](https://mlsanity.com/paper/1703.09210/) (2017)
- [SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1810.06498/) (2018)
- [Generative Image Modeling using Style and Structure Adversarial Networks](https://mlsanity.com/paper/1603.05631/) (2016)
- [Red-GAN: Attacking class imbalance via conditioned generation. Yet another perspective on medical image synthesis for skin lesion dermoscopy and brain tumor MRI](https://mlsanity.com/paper/2004.10734/) (2020)
- [Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets](https://mlsanity.com/paper/2009.04009/) (2020)
- [Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis](https://mlsanity.com/paper/2204.02854/) (2022)

## Built on by (most cited first, 30 of 37)

- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis](https://mlsanity.com/paper/2305.18453/) (2023)
- [Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models](https://mlsanity.com/paper/2402.05210/) (2024)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [Towards Generalizable Tumor Synthesis](https://mlsanity.com/paper/2402.19470/) (2024)
- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation](https://mlsanity.com/paper/2304.04106/) (2023)
- [DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology](https://mlsanity.com/paper/2306.13384/) (2023)
- [Brain tumor segmentation using synthetic MR images -- A comparison of GANs and diffusion models](https://mlsanity.com/paper/2306.02986/) (2023)
- [Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models](https://mlsanity.com/paper/2306.01322/) (2023)
- [AI-Generated Images as Data Source: The Dawn of Synthetic Era](https://mlsanity.com/paper/2310.01830/) (2023)
- [Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data](https://mlsanity.com/paper/2510.14831/) (2025)
- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions](https://mlsanity.com/paper/2506.23440/) (2025)
- [Regression is all you need for medical image translation](https://mlsanity.com/paper/2505.02048/) (2025)
- [Probing the Limits and Capabilities of Diffusion Models for the Anatomic Editing of Digital Twins](https://mlsanity.com/paper/2401.00247/) (2023)
- [A 3D generative model of pathological multi-modal MR images and segmentations](https://mlsanity.com/paper/2311.04552/) (2023)
- [SynCellFactory: Generative Data Augmentation for Cell Tracking](https://mlsanity.com/paper/2404.16421/) (2024)
- [Anatomical feature-prioritized loss for enhanced MR to CT translation](https://mlsanity.com/paper/2410.10328/) (2024)
- [Learning Expected Appearances for Intraoperative Registration during Neurosurgery](https://mlsanity.com/paper/2310.01735/) (2023)
- [Does Training on Synthetic Data Make Models Less Robust?](https://mlsanity.com/paper/2502.07164/) (2025)
- [Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification](https://mlsanity.com/paper/2411.04956/) (2024)
- [S2S2: Semantic Stacking for Robust Semantic Segmentation in Medical Imaging](https://mlsanity.com/paper/2412.13156/) (2024)
- [Bounding Box-Guided Diffusion for Synthesizing Industrial Images and Segmentation Map](https://mlsanity.com/paper/2505.03623/) (2025)
- [Diffusion-Driven Generation of Minimally Preprocessed Brain MRI](https://mlsanity.com/paper/2510.26834/) (2025)
- [Guided Synthesis of Labeled Brain MRI Data Using Latent Diffusion Models for Segmentation of Enlarged Ventricles](https://mlsanity.com/paper/2411.01351/) (2024)
- [Inpainting is All You Need: A Diffusion-based Augmentation Method for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2506.23038/) (2025)
- [CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models](https://mlsanity.com/paper/2509.08015/) (2025)
- [USB: Unified Synthetic Brain Framework for Bidirectional Pathology-Healthy Generation and Editing](https://mlsanity.com/paper/2512.00269/) (2025)

## Abstract (arXiv)

In order to achieve good performance and generalisability, medical image segmentation models should be trained on sizeable datasets with sufficient variability. Due to ethics and governance restrictions, and the costs associated with labelling data, scientific development is often stifled, with models trained and tested on limited data. Data augmentation is often used to artificially increase the variability in the data distribution and improve model generalisability. Recent works have explored deep generative models for image synthesis, as such an approach would enable the generation of an effectively infinite amount of varied data, addressing the generalisability and data access problems. However, many proposed solutions limit the user's control over what is generated. In this work, we propose brainSPADE, a model which combines a synthetic diffusion-based label generator with a semantic image generator. Our model can produce fully synthetic brain labels on-demand, with or without pathology of interest, and then generate a corresponding MRI image of an arbitrary guided style. Experiments show that brainSPADE synthetic data can be used to train segmentation models with performance comparable to that of models trained on real data.