# MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation

arXiv: https://arxiv.org/abs/2304.04106 · HTML: https://mlsanity.com/paper/2304.04106/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 8 Apr 2023.

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks](https://mlsanity.com/paper/1908.02498/) (2019)
- [Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN](https://mlsanity.com/paper/2008.01910/) (2020)
- [AFTer-UNet: Axial Fusion Transformer UNet for Medical Image Segmentation](https://mlsanity.com/paper/2110.10403/) (2021)
- [Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective](https://mlsanity.com/paper/2302.01735/) (2023)
- [Can segmentation models be trained with fully synthetically generated data?](https://mlsanity.com/paper/2209.08256/) (2022)
- [Class-Aware Adversarial Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2201.10737/) (2022)
- [Synthetic Medical Images from Dual Generative Adversarial Networks](https://mlsanity.com/paper/1709.01872/) (2017)
- [Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels](https://mlsanity.com/paper/2209.13476/) (2022)
- [Bootstrapping Semi-supervised Medical Image Segmentation with Anatomical-aware Contrastive Distillation](https://mlsanity.com/paper/2206.02307/) (2022)
- [Learning Implicit Brain MRI Manifolds with Deep Learning](https://mlsanity.com/paper/1801.01847/) (2018)
- [MelanoGANs: High Resolution Skin Lesion Synthesis with GANs](https://mlsanity.com/paper/1804.04338/) (2018)
- [Mask2Lesion: Mask-Constrained Adversarial Skin Lesion Image Synthesis](https://mlsanity.com/paper/1906.05845/) (2019)
- [4D Semantic Cardiac Magnetic Resonance Image Synthesis on XCAT Anatomical Model](https://mlsanity.com/paper/2002.07089/) (2020)

## Built on by (most cited first, 30 of 34)

- [DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models](https://mlsanity.com/paper/2308.06160/) (2023)
- [MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images](https://mlsanity.com/paper/2310.03559/) (2023)
- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges](https://mlsanity.com/paper/2407.00116/) (2024)
- [Generative Enhancement for 3D Medical Images](https://mlsanity.com/paper/2403.12852/) (2024)
- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency](https://mlsanity.com/paper/2507.00802/) (2025)
- [MedDiff-FM: A Diffusion-based Foundation Model for Versatile Medical Image Applications](https://mlsanity.com/paper/2410.15432/) (2024)
- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)
- [Memory-efficient High-resolution OCT Volume Synthesis with Cascaded Amortized Latent Diffusion Models](https://mlsanity.com/paper/2405.16516/) (2024)
- [SynthSet: Generative Diffusion Model for Semantic Segmentation in Precision Agriculture](https://mlsanity.com/paper/2411.03505/) (2024)
- [LCMem: A Universal Model for Robust Image Memorization Detection](https://mlsanity.com/paper/2512.14421/) (2025)
- [Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimes](https://mlsanity.com/paper/2406.11659/) (2024)
- [Enabling PSO-Secure Synthetic Data Sharing Using Diversity-Aware Diffusion Models](https://mlsanity.com/paper/2506.17975/) (2025)
- [The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination](https://mlsanity.com/paper/2601.08237/) (2026)
- [Hyperparameter-Free Medical Image Synthesis for Sharing Data and Improving Site-Specific Segmentation](https://mlsanity.com/paper/2404.06240/) (2024)
- [seg2med: a bridge from artificial anatomy to multimodal medical images](https://mlsanity.com/paper/2504.09182/) (2025)
- [A Diffusion-Driven Temporal Super-Resolution and Spatial Consistency Enhancement Framework for 4D MRI imaging](https://mlsanity.com/paper/2506.04116/) (2025)
- [VAP-Diffusion: Enriching Descriptions with MLLMs for Enhanced Medical Image Generation](https://mlsanity.com/paper/2506.23641/) (2025)
- [Anatomically Guided Latent Diffusion for Brain MRI Progression Modeling](https://mlsanity.com/paper/2601.14584/) (2026)
- [NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction](https://mlsanity.com/paper/2606.24087/) (2026)
- [Guided Synthesis of Labeled Brain MRI Data Using Latent Diffusion Models for Segmentation of Enlarged Ventricles](https://mlsanity.com/paper/2411.01351/) (2024)
- [Inpainting is All You Need: A Diffusion-based Augmentation Method for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2506.23038/) (2025)
- [Integrating Anatomical Priors into a Causal Diffusion Model](https://mlsanity.com/paper/2509.09054/) (2025)
- [Cortex-Grounded Diffusion Models for Brain Image Generation](https://mlsanity.com/paper/2601.19498/) (2026)
- [Knowledge-Guided 3D CT Generation: A Conditioning-Centric Taxonomy](https://mlsanity.com/paper/2608.09992/) (2026)
- [GuideGen: A Text-Guided Framework for Paired Full-torso Anatomy and CT Volume Generation](https://mlsanity.com/paper/2403.07247/) (2024)
- [Multi-Conditioned Denoising Diffusion Probabilistic Model (mDDPM) for Medical Image Synthesis](https://mlsanity.com/paper/2409.04670/) (2024)
- [StructDiff: Structure-aware Diffusion Model for 3D Fine-grained Medical Image Synthesis](https://mlsanity.com/paper/2503.09560/) (2025)
- [Toward Using Machine Learning as a Shape Quality Metric for Liver Point Cloud Generation](https://mlsanity.com/paper/2508.02482/) (2025)

## Abstract (arXiv)

Acquiring and annotating sufficient labeled data is crucial in developing accurate and robust learning-based models, but obtaining such data can be challenging in many medical image segmentation tasks. One promising solution is to synthesize realistic data with ground-truth mask annotations. However, no prior studies have explored generating complete 3D volumetric images with masks. In this paper, we present MedGen3D, a deep generative framework that can generate paired 3D medical images and masks. First, we represent the 3D medical data as 2D sequences and propose the Multi-Condition Diffusion Probabilistic Model (MC-DPM) to generate multi-label mask sequences adhering to anatomical geometry. Then, we use an image sequence generator and semantic diffusion refiner conditioned on the generated mask sequences to produce realistic 3D medical images that align with the generated masks. Our proposed framework guarantees accurate alignment between synthetic images and segmentation maps. Experiments on 3D thoracic CT and brain MRI datasets show that our synthetic data is both diverse and faithful to the original data, and demonstrate the benefits for downstream segmentation tasks. We anticipate that MedGen3D's ability to synthesize paired 3D medical images and masks will prove valuable in training deep learning models for medical imaging tasks.