# 3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation

arXiv: https://arxiv.org/abs/2412.13059 · HTML: https://mlsanity.com/paper/2412.13059/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 50 of its 66 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 50)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems](https://mlsanity.com/paper/2303.05754/) (2023)
- [Mode Regularized Generative Adversarial Networks](https://mlsanity.com/paper/1612.02136/) (2016)
- [MosMedData: Chest CT Scans With COVID-19 Related Findings Dataset](https://mlsanity.com/paper/2005.06465/) (2020)
- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [Diffusion Probabilistic Models beat GANs on Medical Images](https://mlsanity.com/paper/2212.07501/) (2022)

## Built on by (most cited first, 30 of 37)

- [MAISI-v2: Accelerated 3D High-Resolution Medical Image Synthesis with Rectified Flow and Region-specific Contrastive Loss](https://mlsanity.com/paper/2508.05772/) (2025)
- [Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation](https://mlsanity.com/paper/2508.09177/) (2025)
- [Applications of Large Models in Medicine](https://mlsanity.com/paper/2502.17132/) (2025)
- [Latent Diffusion Autoencoders: Toward Efficient and Meaningful Unsupervised Representation Learning in Medical Imaging](https://mlsanity.com/paper/2504.08635/) (2025)
- [Improving 2D Diffusion Models for 3D Medical Imaging with Inter-Slice Consistent Stochasticity](https://mlsanity.com/paper/2602.04162/) (2026)
- [Scalable next-scale autoregression for medical image generation across anatomical regions](https://mlsanity.com/paper/2602.14512/) (2026)
- [TumorFlow: Physics-Guided Longitudinal MRI Synthesis of Glioblastoma Growth](https://mlsanity.com/paper/2603.04058/) (2026)
- [GenMed: A Pairwise Generative Reformulation of Medical Diagnostic Tasks](https://mlsanity.com/paper/2605.10645/) (2026)
- [SynBT: High-quality Tumor Synthesis for Breast Tumor Segmentation by 3D Diffusion Model](https://mlsanity.com/paper/2509.03267/) (2025)
- [MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis](https://mlsanity.com/paper/2602.17901/) (2026)
- [Exploiting Completeness Perception with Diffusion Transformer for Unified 3D MRI Synthesis](https://mlsanity.com/paper/2602.18400/) (2026)
- [GloResNet: A lightweight 3D CNN with global topological features for preterm brain injury prediction](https://mlsanity.com/paper/2606.02498/) (2026)
- [WaveDiT: Distribution-Aware Wavelet Flow Matching for Efficient 3D Brain MRI Synthesis](https://mlsanity.com/paper/2606.08670/) (2026)
- [BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation](https://mlsanity.com/paper/2606.19651/) (2026)
- [Beyond Representation Learning: A Systematic Study of Joint-Embedding Predictive Generation for 3D Brain MRI](https://mlsanity.com/paper/2608.28787/) (2026)
- [TAlignDiff: Automatic Tooth Alignment assisted by Diffusion-based Transformation Learning](https://mlsanity.com/paper/2508.04565/) (2025)
- [Spatio-Temporal Conditional Diffusion Models for Forecasting Future Multiple Sclerosis Lesion Masks Conditioned on Treatments](https://mlsanity.com/paper/2508.07006/) (2025)
- [K-Syn: K-space Data Synthesis in Ultra Low-data Regimes](https://mlsanity.com/paper/2509.06997/) (2025)
- [ALL-PET: A Low-resource and Low-shot PET Foundation Model in Projection Domain](https://mlsanity.com/paper/2509.09130/) (2025)
- [BrainNormalizer: Anatomy-Informed Pseudo-Healthy Brain Reconstruction from Tumor MRI via Edge-Guided ControlNet](https://mlsanity.com/paper/2511.12853/) (2025)
- [Joint 3D Gravity and Magnetic Inversion via Rectified Flow and Ginzburg-Landau Guidance](https://mlsanity.com/paper/2603.06829/) (2026)
- [VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers](https://mlsanity.com/paper/2603.25181/) (2026)
- [CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis](https://mlsanity.com/paper/2603.25194/) (2026)
- [PHASOR: Anatomy- and Phase-Consistent Volumetric Diffusion for CT Virtual Contrast Enhancement](https://mlsanity.com/paper/2604.01053/) (2026)
- [Few-Shot Distribution-Aligned Flow Matching for Data Synthesis in Medical Image Segmentation](https://mlsanity.com/paper/2604.02868/) (2026)
- [Distilling Photon-Counting CT into Routine Chest CT through Clinically Validated Degradation Modeling](https://mlsanity.com/paper/2604.07329/) (2026)
- [Weakly-Supervised Lung Nodule Segmentation via Training-Free Guidance of 3D Rectified Flow](https://mlsanity.com/paper/2604.08313/) (2026)
- [LiFT: Lifted Inter-slice Feature Trajectories for 3D Image Generation from 2D Generators](https://mlsanity.com/paper/2605.19060/) (2026)
- [MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts](https://mlsanity.com/paper/2606.00967/) (2026)
- [Multiscale Hypersonic Boundary Layer Reconstruction via Spectral Binning and Subdomain-wise Conditional Diffusion](https://mlsanity.com/paper/2606.15023/) (2026)

## Abstract (arXiv)

The generation of medical images presents significant challenges due to their high-resolution and three-dimensional nature. Existing methods often yield suboptimal performance in generating high-quality 3D medical images, and there is currently no universal generative framework for medical imaging. In this paper, we introduce a 3D Medical Latent Diffusion (3D MedDiffusion) model for controllable, high-quality 3D medical image generation. 3D MedDiffusion incorporates a novel, highly efficient Patch-Volume Autoencoder that compresses medical images into latent space through patch-wise encoding and recovers back into image space through volume-wise decoding. Additionally, we design a new noise estimator to capture both local details and global structural information during diffusion denoising process. 3D MedDiffusion can generate fine-detailed, high-resolution images (up to 512x512x512) and effectively adapt to various downstream tasks as it is trained on large-scale datasets covering CT and MRI modalities and different anatomical regions (from head to leg). Experimental results demonstrate that 3D MedDiffusion surpasses state-of-the-art methods in generative quality and exhibits strong generalizability across tasks such as sparse-view CT reconstruction, fast MRI reconstruction, and data augmentation for segmentation and classification. Source code and checkpoints are available at https://github.com/ShanghaiTech-IMPACT/3D-MedDiffusion.