# Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis

arXiv: https://arxiv.org/abs/2307.10094 · HTML: https://mlsanity.com/paper/2307.10094/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2211.10655/) (2022)
- [Diffusion Deformable Model for 4D Temporal Medical Image Generation](https://mlsanity.com/paper/2206.13295/) (2022)
- [Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Generation](https://mlsanity.com/paper/2211.03364/) (2022)
- [Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models](https://mlsanity.com/paper/2303.08440/) (2023)
- [Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection](https://mlsanity.com/paper/1802.07846/) (2018)
- [Memory-efficient GAN-based Domain Translation of High Resolution 3D Medical Images](https://mlsanity.com/paper/2010.03396/) (2020)
- [On the Implicit Assumptions of GANs](https://mlsanity.com/paper/1811.12402/) (2018)

## Built on by (most cited first, 30 of 43)

- [WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis](https://mlsanity.com/paper/2402.19043/) (2024)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [Denoising Diffusion Models for 3D Healthy Brain Tissue Inpainting](https://mlsanity.com/paper/2403.14499/) (2024)
- [Similarity and Quality Metrics for MR Image-To-Image Translation](https://mlsanity.com/paper/2405.08431/) (2024)
- [MAISI-v2: Accelerated 3D High-Resolution Medical Image Synthesis with Rectified Flow and Region-specific Contrastive Loss](https://mlsanity.com/paper/2508.05772/) (2025)
- [Slice-Consistent 3D Volumetric Brain CT-to-MRI Translation with 2D Brownian Bridge Diffusion Model](https://mlsanity.com/paper/2407.05059/) (2024)
- [Generative Enhancement for 3D Medical Images](https://mlsanity.com/paper/2403.12852/) (2024)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Medical Vision Generalist: Unifying Medical Imaging Tasks in Context](https://mlsanity.com/paper/2406.05565/) (2024)
- [PASTA: Pathology-Aware MRI to PET Cross-Modal Translation with Diffusion Models](https://mlsanity.com/paper/2405.16942/) (2024)
- [DIFR3CT: Latent Diffusion for Probabilistic 3D CT Reconstruction from Few Planar X-Rays](https://mlsanity.com/paper/2408.15118/) (2024)
- [TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency](https://mlsanity.com/paper/2507.00802/) (2025)
- [Lung-DDPM: Semantic Layout-guided Diffusion Models for Thoracic CT Image Synthesis](https://mlsanity.com/paper/2502.15204/) (2025)
- [Physics-Inspired Generative Models in Medical Imaging: A Review](https://mlsanity.com/paper/2407.10856/) (2024)
- [Unpaired Volumetric Harmonization of Brain MRI with Conditional Latent Diffusion](https://mlsanity.com/paper/2408.09315/) (2024)
- [Blaze3DM: Marry Triplane Representation with Diffusion for 3D Medical Inverse Problem Solving](https://mlsanity.com/paper/2405.15241/) (2024)
- [McCaD: Multi-Contrast MRI Conditioned, Adaptive Adversarial Diffusion Model for High-Fidelity MRI Synthesis](https://mlsanity.com/paper/2409.00585/) (2024)
- [Cross-modality image synthesis from TOF-MRA to CTA using diffusion-based models](https://mlsanity.com/paper/2409.10089/) (2024)
- [Anatomical feature-prioritized loss for enhanced MR to CT translation](https://mlsanity.com/paper/2410.10328/) (2024)
- [Brain Tumour Removing and Missing Modality Generation using 3D WDM](https://mlsanity.com/paper/2411.04630/) (2024)
- [Denoising Diffusion Models for Anomaly Localization in Medical Images](https://mlsanity.com/paper/2410.23834/) (2024)
- [X-ray2CTPA: Leveraging Diffusion Models to Enhance Pulmonary Embolism Classification](https://mlsanity.com/paper/2406.16109/) (2024)
- [LPUWF-LDM: Enhanced Latent Diffusion Model for Precise Late-phase UWF-FA Generation on Limited Dataset](https://mlsanity.com/paper/2409.00726/) (2024)
- [UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model](https://mlsanity.com/paper/2507.18362/) (2025)
- [Early Lung Cancer Diagnosis from Virtual Follow-up LDCT Generation via Correlational Autoencoder and Latent Flow Matching](https://mlsanity.com/paper/2511.18185/) (2025)
- [Cascaded 3D Diffusion Models for Whole-body 3D 18-F FDG PET/CT synthesis from Demographics](https://mlsanity.com/paper/2505.22489/) (2025)
- [3D Conditional Image Synthesis of Left Atrial LGE MRI from Composite Semantic Masks](https://mlsanity.com/paper/2601.04588/) (2026)
- [Translating MRI to PET through Conditional Diffusion Models with Enhanced Pathology Awareness](https://mlsanity.com/paper/2603.18896/) (2026)
- [Diffusion Model-based FOD Restoration from High Distortion in dMRI](https://mlsanity.com/paper/2406.13209/) (2024)

## Abstract (arXiv)

Cross-modality medical image synthesis is a critical topic and has the potential to facilitate numerous applications in the medical imaging field. Despite recent successes in deep-learning-based generative models, most current medical image synthesis methods rely on generative adversarial networks and suffer from notorious mode collapse and unstable training. Moreover, the 2D backbone-driven approaches would easily result in volumetric inconsistency, while 3D backbones are challenging and impractical due to the tremendous memory cost and training difficulty. In this paper, we introduce a new paradigm for volumetric medical data synthesis by leveraging 2D backbones and present a diffusion-based framework, Make-A-Volume, for cross-modality 3D medical image synthesis. To learn the cross-modality slice-wise mapping, we employ a latent diffusion model and learn a low-dimensional latent space, resulting in high computational efficiency. To enable the 3D image synthesis and mitigate volumetric inconsistency, we further insert a series of volumetric layers in the 2D slice-mapping model and fine-tune them with paired 3D data. This paradigm extends the 2D image diffusion model to a volumetric version with a slightly increasing number of parameters and computation, offering a principled solution for generic cross-modality 3D medical image synthesis. We showcase the effectiveness of our Make-A-Volume framework on an in-house SWI-MRA brain MRI dataset and a public T1-T2 brain MRI dataset. Experimental results demonstrate that our framework achieves superior synthesis results with volumetric consistency.