# Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray images

arXiv: https://arxiv.org/abs/2305.07644 · HTML: https://mlsanity.com/paper/2305.07644/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 12 May 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images](https://mlsanity.com/paper/2209.13167/) (2022)
- [Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems](https://mlsanity.com/paper/2211.01323/) (2022)
- [Spot the fake lungs: Generating Synthetic Medical Images using Neural Diffusion Models](https://mlsanity.com/paper/2211.00902/) (2022)
- [Ensembles of GANs for synthetic training data generation](https://mlsanity.com/paper/2104.11797/) (2021)
- [On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity](https://mlsanity.com/paper/2205.12642/) (2022)
- [Does an ensemble of GANs lead to better performance when training segmentation networks with synthetic images?](https://mlsanity.com/paper/2211.04086/) (2022)

## Built on by (most cited first, 23 of 23)

- [A Geometric Framework for Understanding Memorization in Generative Models](https://mlsanity.com/paper/2411.00113/) (2024)
- [Brain tumor segmentation using synthetic MR images -- A comparison of GANs and diffusion models](https://mlsanity.com/paper/2306.02986/) (2023)
- [Metrics that matter: Evaluating image quality metrics for medical image generation](https://mlsanity.com/paper/2505.07175/) (2025)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Assessing the capacity of a denoising diffusion probabilistic model to reproduce spatial context](https://mlsanity.com/paper/2309.10817/) (2023)
- [Boosting Data Analytics With Synthetic Volume Expansion](https://mlsanity.com/paper/2310.17848/) (2023)
- [U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AI](https://mlsanity.com/paper/2406.15386/) (2024)
- [CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs](https://mlsanity.com/paper/2505.10496/) (2025)
- [MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection](https://mlsanity.com/paper/2405.19458/) (2024)
- [Benchmarking GANs, Diffusion Models, and Flow Matching for T1w-to-T2w MRI Translation](https://mlsanity.com/paper/2507.14575/) (2025)
- [The Devil is in the Prompts: De-Identification Traces Enhance Memorization Risks in Synthetic Chest X-Ray Generation](https://mlsanity.com/paper/2502.07516/) (2025)
- [Frequency-Calibrated Membership Inference Attacks on Medical Image Diffusion Models](https://mlsanity.com/paper/2506.14919/) (2025)
- [Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches](https://mlsanity.com/paper/2506.10825/) (2025)
- [Can Diffusion Models Bridge the Domain Gap in Cardiac MR Imaging?](https://mlsanity.com/paper/2508.06327/) (2025)
- [Transport Novelty Distance: A Distributional Metric for Evaluating Material Generative Models](https://mlsanity.com/paper/2512.09514/) (2025)
- [TumorFlow: Physics-Guided Longitudinal MRI Synthesis of Glioblastoma Growth](https://mlsanity.com/paper/2603.04058/) (2026)
- [Beyond the Generative Learning Trilemma: Generative Model Assessment in Data Scarcity Domains](https://mlsanity.com/paper/2504.10555/) (2025)
- [Cross Modality Image Translation In Medical Imaging Using Generative Frameworks](https://mlsanity.com/paper/2605.13686/) (2026)
- [Generative artificial intelligence in ophthalmology: multimodal retinal images for the diagnosis of Alzheimer's disease with convolutional neural networks](https://mlsanity.com/paper/2406.18247/) (2024)
- [Learning and Generating Diverse Residential Load Patterns Using GAN with Weakly-Supervised Training and Weight Selection](https://mlsanity.com/paper/2504.14300/) (2025)
- [SkinGenBench: Generative Model and Preprocessing Effects for Synthetic Dermoscopic Augmentation in Melanoma Diagnosis](https://mlsanity.com/paper/2512.17585/) (2025)
- [A Calibrated Memorization Index (MI) for Detecting Training Data Leakage in Generative MRI Models](https://mlsanity.com/paper/2602.13066/) (2026)
- [HUydra: Full-Range Lung CT Synthesis via Multiple HU Interval Generative Modelling](https://mlsanity.com/paper/2603.23041/) (2026)

## Abstract (arXiv)

Diffusion models were initially developed for text-to-image generation and are now being utilized to generate high quality synthetic images. Preceded by GANs, diffusion models have shown impressive results using various evaluation metrics. However, commonly used metrics such as FID and IS are not suitable for determining whether diffusion models are simply reproducing the training images. Here we train StyleGAN and a diffusion model, using BRATS20, BRATS21 and a chest x-ray pneumonia dataset, to synthesize brain MRI and chest x-ray images, and measure the correlation between the synthetic images and all training images. Our results show that diffusion models are more likely to memorize the training images, compared to StyleGAN, especially for small datasets and when using 2D slices from 3D volumes. Researchers should be careful when using diffusion models (and to some extent GANs) for medical imaging, if the final goal is to share the synthetic images.