# Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesis

arXiv: https://arxiv.org/abs/2409.13532 · HTML: https://mlsanity.com/paper/2409.13532/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 14 of 14)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities](https://mlsanity.com/paper/2111.04735/) (2021)
- [MR-Contrast-Aware Image-to-Image Translations with Generative Adversarial Networks](https://mlsanity.com/paper/2104.01449/) (2021)
- [A Tool for Super-Resolving Multimodal Clinical MRI](https://mlsanity.com/paper/1909.01140/) (2019)
- [Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery](https://mlsanity.com/paper/2202.03242/) (2022)
- [Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environments](https://mlsanity.com/paper/2211.03587/) (2022)
- [Generalizable synthetic MRI with physics-informed convolutional networks](https://mlsanity.com/paper/2305.12570/) (2023)

## Built on by (most cited first, 7 of 7)

- [Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis](https://mlsanity.com/paper/2412.20651/) (2024)
- [VISAGE: Video Synthesis using Action Graphs for Surgery](https://mlsanity.com/paper/2410.17751/) (2024)
- [LLMs Help Alleviate the Cross-Subject Variability in Brain Signal and Language Alignment](https://mlsanity.com/paper/2501.02621/) (2025)
- [Diffusion Models for conditional MRI generation](https://mlsanity.com/paper/2502.18620/) (2025)
- [Anatomically Guided Latent Diffusion for Brain MRI Progression Modeling](https://mlsanity.com/paper/2601.14584/) (2026)
- [Mitigating 3D Prostate Biparametric MRI Data Scarcity through Domain Adaptation using Locally-Trained Latent Diffusion Models for Prostate Cancer Detection](https://mlsanity.com/paper/2507.06384/) (2025)
- [Quantitative mapping from conventional MRI using self-supervised physics-guided deep learning: applications to a large-scale, clinically heterogeneous dataset](https://mlsanity.com/paper/2601.05063/) (2026)

## Abstract (arXiv)

Recent advances in generative models for medical imaging have shown promise in representing multiple modalities. However, the variability in modality availability across datasets limits the general applicability of the synthetic data they produce. To address this, we present a novel physics-informed generative model capable of synthesizing a variable number of brain MRI modalities, including those not present in the original dataset. Our approach utilizes latent diffusion models and a two-step generative process: first, unobserved physical tissue property maps are synthesized using a latent diffusion model, and then these maps are combined with a physical signal model to generate the final MRI scan. Our experiments demonstrate the efficacy of this approach in generating unseen MR contrasts and preserving physical plausibility. Furthermore, we validate the distributions of generated tissue properties by comparing them to those measured in real brain tissue.