# CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesis

arXiv: https://arxiv.org/abs/2303.14081 · HTML: https://mlsanity.com/paper/2303.14081/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [On Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.00132/) (2021)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Attentional Feature Fusion](https://mlsanity.com/paper/2009.14082/) (2020)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models](https://mlsanity.com/paper/2209.12104/) (2022)
- [DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis](https://mlsanity.com/paper/1904.12894/) (2019)
- [A Closer Look at the Optimization Landscapes of Generative Adversarial Networks](https://mlsanity.com/paper/1906.04848/) (2019)
- [Multi-modal learning for predicting the genotype of glioma](https://mlsanity.com/paper/2203.10852/) (2022)
- [Progressively Volumetrized Deep Generative Models for Data-Efficient Contextual Learning of MR Image Recovery](https://mlsanity.com/paper/2011.13913/) (2020)
- [MouseGAN++: Unsupervised Disentanglement and Contrastive Representation for Multiple MRI Modalities Synthesis and Structural Segmentation of Mouse Brain](https://mlsanity.com/paper/2212.01825/) (2022)

## Built on by (most cited first, 30 of 65)

- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2404.04785/) (2024)
- [DiffBoost: Enhancing Medical Image Segmentation via Text-Guided Diffusion Model](https://mlsanity.com/paper/2310.12868/) (2023)
- [What You Have is What You Track: Adaptive and Robust Multimodal Tracking](https://mlsanity.com/paper/2507.05899/) (2025)
- [Generative Enhancement for 3D Medical Images](https://mlsanity.com/paper/2403.12852/) (2024)
- [Metrics that matter: Evaluating image quality metrics for medical image generation](https://mlsanity.com/paper/2505.07175/) (2025)
- [Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning](https://mlsanity.com/paper/2403.06516/) (2024)
- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts](https://mlsanity.com/paper/2410.19378/) (2024)
- [Phy-Diff: Physics-guided Hourglass Diffusion Model for Diffusion MRI Synthesis](https://mlsanity.com/paper/2406.03002/) (2024)
- [SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models](https://mlsanity.com/paper/2412.02332/) (2024)
- [Regression is all you need for medical image translation](https://mlsanity.com/paper/2505.02048/) (2025)
- [TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency](https://mlsanity.com/paper/2507.00802/) (2025)
- [3D MRI Synthesis with Slice-Based Latent Diffusion Models: Improving Tumor Segmentation Tasks in Data-Scarce Regimes](https://mlsanity.com/paper/2406.05421/) (2024)
- [Shape-guided Conditional Latent Diffusion Models for Synthesising Brain Vasculature](https://mlsanity.com/paper/2308.06781/) (2023)
- [Unpaired Volumetric Harmonization of Brain MRI with Conditional Latent Diffusion](https://mlsanity.com/paper/2408.09315/) (2024)
- [Diffusion Models for Computational Neuroimaging: A Survey](https://mlsanity.com/paper/2502.06552/) (2025)
- [FetalFlex: Anatomy-Guided Diffusion Model for Flexible Control on Fetal Ultrasound Image Synthesis](https://mlsanity.com/paper/2503.14906/) (2025)
- [A Unified Framework for Generative Data Augmentation: A Comprehensive Survey](https://mlsanity.com/paper/2310.00277/) (2023)
- [Angio-Diff: Learning a Self-Supervised Adversarial Diffusion Model for Angiographic Geometry Generation](https://mlsanity.com/paper/2506.19455/) (2025)
- [LesionGen: A Concept-Guided Diffusion Model for Dermatology Image Synthesis](https://mlsanity.com/paper/2507.23001/) (2025)
- [CAVM: Conditional Autoregressive Vision Model for Contrast-Enhanced Brain Tumor MRI Synthesis](https://mlsanity.com/paper/2406.16074/) (2024)
- [FgC2F-UDiff: Frequency-guided and Coarse-to-fine Unified Diffusion Model for Multi-modality Missing MRI Synthesis](https://mlsanity.com/paper/2501.03526/) (2025)
- [3D Wavelet Latent Diffusion Model for Whole-Body MR-to-CT Modality Translation](https://mlsanity.com/paper/2507.11557/) (2025)
- [MCAD: Multi-modal Conditioned Adversarial Diffusion Model for High-Quality PET Image Reconstruction](https://mlsanity.com/paper/2406.13150/) (2024)
- [I2I-Galip: Unsupervised Medical Image Translation Using Generative Adversarial CLIP](https://mlsanity.com/paper/2409.12399/) (2024)
- [HC3L-Diff: Hybrid conditional latent diffusion with high frequency enhancement for CBCT-to-CT synthesis](https://mlsanity.com/paper/2411.01575/) (2024)
- [Sub-DM:Subspace Diffusion Model with Orthogonal Decomposition for MRI Reconstruction](https://mlsanity.com/paper/2411.03758/) (2024)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)

## Abstract (arXiv)

MRI synthesis promises to mitigate the challenge of missing MRI modality in clinical practice. Diffusion model has emerged as an effective technique for image synthesis by modelling complex and variable data distributions. However, most diffusion-based MRI synthesis models are using a single modality. As they operate in the original image domain, they are memory-intensive and less feasible for multi-modal synthesis. Moreover, they often fail to preserve the anatomical structure in MRI. Further, balancing the multiple conditions from multi-modal MRI inputs is crucial for multi-modal synthesis. Here, we propose the first diffusion-based multi-modality MRI synthesis model, namely Conditioned Latent Diffusion Model (CoLa-Diff). To reduce memory consumption, we design CoLa-Diff to operate in the latent space. We propose a novel network architecture, e.g., similar cooperative filtering, to solve the possible compression and noise in latent space. To better maintain the anatomical structure, brain region masks are introduced as the priors of density distributions to guide diffusion process. We further present auto-weight adaptation to employ multi-modal information effectively. Our experiments demonstrate that CoLa-Diff outperforms other state-of-the-art MRI synthesis methods, promising to serve as an effective tool for multi-modal MRI synthesis.