# Using Diffusion Models to Generate Synthetic Labelled Data for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2310.16794 · HTML: https://mlsanity.com/paper/2310.16794/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 18 of 18)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [Diffusion-based Image Translation using Disentangled Style and Content Representation](https://mlsanity.com/paper/2209.15264/) (2022)
- [Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images](https://mlsanity.com/paper/2106.07009/) (2021)
- [Differentially Private Diffusion Models Generate Useful Synthetic Images](https://mlsanity.com/paper/2302.13861/) (2023)
- [A multi-centre polyp detection and segmentation dataset for generalisability assessment](https://mlsanity.com/paper/2106.04463/) (2021)
- [Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer](https://mlsanity.com/paper/2303.08622/) (2023)
- [Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2](https://mlsanity.com/paper/2210.00586/) (2022)
- [SinGAN-Seg: Synthetic training data generation for medical image segmentation](https://mlsanity.com/paper/2107.00471/) (2021)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)
- [ArSDM: Colonoscopy Images Synthesis with Adaptive Refinement Semantic Diffusion Models](https://mlsanity.com/paper/2309.01111/) (2023)
- [Shape-Aware Complementary-Task Learning for Multi-Organ Segmentation](https://mlsanity.com/paper/1908.05099/) (2019)
- [DivergentNets: Medical Image Segmentation by Network Ensemble](https://mlsanity.com/paper/2107.00283/) (2021)
- [Deploying a Top-100 Supercomputer for Large Parallel Workloads: the Niagara Supercomputer](https://mlsanity.com/paper/1907.13600/) (2019)

## Built on by (most cited first, 10 of 10)

- [The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better](https://mlsanity.com/paper/2406.05184/) (2024)
- [HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2506.21287/) (2025)
- [The Devil is in the Prompts: De-Identification Traces Enhance Memorization Risks in Synthetic Chest X-Ray Generation](https://mlsanity.com/paper/2502.07516/) (2025)
- [ViCTr: Vital Consistency Transfer for Pathology Aware Image Synthesis](https://mlsanity.com/paper/2505.04963/) (2025)
- [Understanding Trade offs When Conditioning Synthetic Data](https://mlsanity.com/paper/2507.02217/) (2025)
- [SRA-Seg: Synthetic to Real Alignment for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2602.02944/) (2026)
- [Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function](https://mlsanity.com/paper/2408.00707/) (2024)
- [TOM: An Open-Source Tongue Segmentation Method with Multi-Teacher Distillation and Task-Specific Data Augmentation](https://mlsanity.com/paper/2508.14932/) (2025)
- [Knowledge-based anomaly detection for identifying network-induced shape artifacts](https://mlsanity.com/paper/2511.04729/) (2025)
- [AWDiff: An a trous wavelet diffusion model for lung ultrasound image synthesis](https://mlsanity.com/paper/2603.03125/) (2026)

## Abstract (arXiv)

Medical image analysis has become a prominent area where machine learning has been applied. However, high quality, publicly available data is limited either due to patient privacy laws or the time and cost required for experts to annotate images. In this retrospective study, we designed and evaluated a pipeline to generate synthetic labeled polyp images for augmenting medical image segmentation models with the aim of reducing this data scarcity. In particular, we trained diffusion models on the HyperKvasir dataset, comprising 1000 images of polyps in the human GI tract from 2008 to 2016. Qualitative expert review, Fr'echet Inception Distance (FID), and Multi-Scale Structural Similarity (MS-SSIM) were tested for evaluation. Additionally, various segmentation models were trained with the generated data and evaluated using Dice score and Intersection over Union. We found that our pipeline produced images more akin to real polyp images based on FID scores, and segmentation performance also showed improvements over GAN methods when trained entirely, or partially, with synthetic data, despite requiring less compute for training. Moreover, the improvement persists when tested on different datasets, showcasing the transferability of the generated images.