# On Differentially Private 3D Medical Image Synthesis with Controllable Latent Diffusion Models

arXiv: https://arxiv.org/abs/2407.16405 · HTML: https://mlsanity.com/paper/2407.16405/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Differentially Private Diffusion Models Generate Useful Synthetic Images](https://mlsanity.com/paper/2302.13861/) (2023)
- [Diffusion Probabilistic Models beat GANs on Medical Images](https://mlsanity.com/paper/2212.07501/) (2022)
- [Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems](https://mlsanity.com/paper/2211.01323/) (2022)
- [Local Differential Privacy: a tutorial](https://mlsanity.com/paper/1907.11908/) (2019)
- [Fairness-Aware Data Augmentation for Cardiac MRI using Text-Conditioned Diffusion Models](https://mlsanity.com/paper/2403.19508/) (2024)
- [Local Differential Privacy Image Generation Using Flow-based Deep Generative Models](https://mlsanity.com/paper/2212.10688/) (2022)

## Built on by (most cited first, 7 of 7)

- [Differential privacy for medical deep learning: methods, tradeoffs, and deployment implications](https://mlsanity.com/paper/2506.00660/) (2025)
- [Frequency-Calibrated Membership Inference Attacks on Medical Image Diffusion Models](https://mlsanity.com/paper/2506.14919/) (2025)
- [One-shot synthesis of rare gastrointestinal lesions improves diagnostic accuracy and clinical training](https://mlsanity.com/paper/2512.24278/) (2025)
- [HUydra: Full-Range Lung CT Synthesis via Multiple HU Interval Generative Modelling](https://mlsanity.com/paper/2603.23041/) (2026)
- [FedDP-PALD: A Privacy-Preserving Federated Latent Diffusion Framework with Prototype Aggregation for Medical Data Synthesis](https://mlsanity.com/paper/2607.16300/) (2026)
- [Semi-Supervised Conditional Diffusion via Label Augmentation](https://mlsanity.com/paper/2607.16685/) (2026)
- [Differentially Private Paired Table-Image Multimodal Synthesis](https://mlsanity.com/paper/2609.00708/) (2026)

## Abstract (arXiv)

Generally, the small size of public medical imaging datasets coupled with stringent privacy concerns, hampers the advancement of data-hungry deep learning models in medical imaging. This study addresses these challenges for 3D cardiac MRI images in the short-axis view. We propose Latent Diffusion Models that generate synthetic images conditioned on medical attributes, while ensuring patient privacy through differentially private model training. To our knowledge, this is the first work to apply and quantify differential privacy in 3D medical image generation. We pre-train our models on public data and finetune them with differential privacy on the UK Biobank dataset. Our experiments reveal that pre-training significantly improves model performance, achieving a Fr'echet Inception Distance (FID) of 26.77 at ε=10, compared to 92.52 for models without pre-training. Additionally, we explore the trade-off between privacy constraints and image quality, investigating how tighter privacy budgets affect output controllability and may lead to degraded performance. Our results demonstrate that proper consideration during training with differential privacy can substantially improve the quality of synthetic cardiac MRI images, but there are still notable challenges in achieving consistent medical realism.