# Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models

arXiv: https://arxiv.org/abs/2402.05210 · HTML: https://mlsanity.com/paper/2402.05210/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Diffusion Models for Medical Anomaly Detection](https://mlsanity.com/paper/2203.04306/) (2022)
- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [Pretraining is All You Need for Image-to-Image Translation](https://mlsanity.com/paper/2205.12952/) (2022)
- [Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models](https://mlsanity.com/paper/2206.03461/) (2022)
- [Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models](https://mlsanity.com/paper/2209.12104/) (2022)
- [PET image denoising based on denoising diffusion probabilistic models](https://mlsanity.com/paper/2209.06167/) (2022)
- [Unsupervised Domain Adaptation via Disentangled Representations: Application to Cross-Modality Liver Segmentation](https://mlsanity.com/paper/1907.13590/) (2019)
- [Can segmentation models be trained with fully synthetically generated data?](https://mlsanity.com/paper/2209.08256/) (2022)
- [Reverse Engineering Breast MRIs: Predicting Acquisition Parameters Directly from Images](https://mlsanity.com/paper/2303.04911/) (2023)
- [Deep Learning for Breast MRI Style Transfer with Limited Training Data](https://mlsanity.com/paper/2301.02069/) (2023)
- [A 3D generative model of pathological multi-modal MR images and segmentations](https://mlsanity.com/paper/2311.04552/) (2023)

## Built on by (most cited first, 30 of 69)

- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [vesselFM: A Foundation Model for Universal 3D Blood Vessel Segmentation](https://mlsanity.com/paper/2411.17386/) (2024)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Towards Learning Contrast Kinetics with Multi-Condition Latent Diffusion Models](https://mlsanity.com/paper/2403.13890/) (2024)
- [PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions](https://mlsanity.com/paper/2506.23440/) (2025)
- [XReal: Realistic Anatomy and Pathology-Aware X-ray Generation via Controllable Diffusion Model](https://mlsanity.com/paper/2403.09240/) (2024)
- [TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency](https://mlsanity.com/paper/2507.00802/) (2025)
- [Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation](https://mlsanity.com/paper/2508.09177/) (2025)
- [Simulating Dynamic Tumor Contrast Enhancement in Breast MRI using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2409.18872/) (2024)
- [Toward Medical Deepfake Detection: A Comprehensive Dataset and Novel Method](https://mlsanity.com/paper/2509.15711/) (2025)
- [Physics-Inspired Generative Models in Medical Imaging: A Review](https://mlsanity.com/paper/2407.10856/) (2024)
- [Generalizable Single-Source Cross-modality Medical Image Segmentation via Invariant Causal Mechanisms](https://mlsanity.com/paper/2411.05223/) (2024)
- [No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-k Patch Sampling](https://mlsanity.com/paper/2501.10814/) (2025)
- [Rethinking Perceptual Metrics for Medical Image Translation](https://mlsanity.com/paper/2404.07318/) (2024)
- [BrainMRDiff: A Diffusion Model for Anatomically Consistent Brain MRI Synthesis](https://mlsanity.com/paper/2504.04532/) (2025)
- [Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology](https://mlsanity.com/paper/2509.17847/) (2025)
- [MRGen: Segmentation Data Engine for Underrepresented MRI Modalities](https://mlsanity.com/paper/2412.04106/) (2024)
- [Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation](https://mlsanity.com/paper/2506.20449/) (2025)
- [Diffusion-Based User-Guided Data Augmentation for Coronary Stenosis Detection](https://mlsanity.com/paper/2508.00438/) (2025)
- [Comparing Conditional Diffusion Models for Synthesizing Contrast-Enhanced Breast MRI from Pre-Contrast Images](https://mlsanity.com/paper/2508.13776/) (2025)
- [Latent Interpolation Learning Using Diffusion Models for Cardiac Volume Reconstruction](https://mlsanity.com/paper/2508.13826/) (2025)
- [Mechanistic Learning with Guided Diffusion Models to Predict Spatio-Temporal Brain Tumor Growth](https://mlsanity.com/paper/2509.09610/) (2025)
- [ContourDiff: Unpaired Medical Image Translation with Structural Consistency](https://mlsanity.com/paper/2403.10786/) (2024)
- [Label-Efficient Data Augmentation with Video Diffusion Models for Guidewire Segmentation in Cardiac Fluoroscopy](https://mlsanity.com/paper/2412.16050/) (2024)
- [LesiOnTime -- Joint Temporal and Clinical Modeling for Small Breast Lesion Segmentation in Longitudinal DCE-MRI](https://mlsanity.com/paper/2508.00496/) (2025)
- [RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion](https://mlsanity.com/paper/2510.14962/) (2025)
- [Diffusion-Driven Generation of Minimally Preprocessed Brain MRI](https://mlsanity.com/paper/2510.26834/) (2025)
- [Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentation](https://mlsanity.com/paper/2412.08116/) (2024)
- [Physics-informed DeepCT: Sinogram Wavelet Decomposition Meets Masked Diffusion](https://mlsanity.com/paper/2501.09935/) (2025)

## Abstract (arXiv)

Diffusion models have enabled remarkably high-quality medical image generation, yet it is challenging to enforce anatomical constraints in generated images. To this end, we propose a diffusion model-based method that supports anatomically-controllable medical image generation, by following a multi-class anatomical segmentation mask at each sampling step. We additionally introduce a random mask ablation training algorithm to enable conditioning on a selected combination of anatomical constraints while allowing flexibility in other anatomical areas. We compare our method ("SegGuidedDiff") to existing methods on breast MRI and abdominal/neck-to-pelvis CT datasets with a wide range of anatomical objects. Results show that our method reaches a new state-of-the-art in the faithfulness of generated images to input anatomical masks on both datasets, and is on par for general anatomical realism. Finally, our model also enjoys the extra benefit of being able to adjust the anatomical similarity of generated images to real images of choice through interpolation in its latent space. SegGuidedDiff has many applications, including cross-modality translation, and the generation of paired or counterfactual data. Our code is available at https://github.com/mazurowski-lab/segmentation-guided-diffusion.