# Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study

arXiv: https://arxiv.org/abs/2311.00265 · HTML: https://mlsanity.com/paper/2311.00265/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2003.00273/) (2020)
- [Brain Tumor Segmentation on MRI with Missing Modalities](https://mlsanity.com/paper/1904.07290/) (2019)
- [Brain tumor segmentation with missing modalities via latent multi-source correlation representation](https://mlsanity.com/paper/2003.08870/) (2020)

## Built on by (most cited first, 30 of 64)

- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [Fast-DDPM: Fast Denoising Diffusion Probabilistic Models for Medical Image-to-Image Generation](https://mlsanity.com/paper/2405.14802/) (2024)
- [Slice-Consistent 3D Volumetric Brain CT-to-MRI Translation with 2D Brownian Bridge Diffusion Model](https://mlsanity.com/paper/2407.05059/) (2024)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models](https://mlsanity.com/paper/2412.02332/) (2024)
- [Regression is all you need for medical image translation](https://mlsanity.com/paper/2505.02048/) (2025)
- [Deep learning for automated detection of breast cancer in deep ultraviolet fluorescence images with diffusion probabilistic model](https://mlsanity.com/paper/2407.00967/) (2024)
- [RadiomicsFill-Mammo: Synthetic Mammogram Mass Manipulation with Radiomics Features](https://mlsanity.com/paper/2407.05683/) (2024)
- [Physics-Inspired Generative Models in Medical Imaging: A Review](https://mlsanity.com/paper/2407.10856/) (2024)
- [Diffusion Models for Computational Neuroimaging: A Survey](https://mlsanity.com/paper/2502.06552/) (2025)
- [Blaze3DM: Marry Triplane Representation with Diffusion for 3D Medical Inverse Problem Solving](https://mlsanity.com/paper/2405.15241/) (2024)
- [MuTri: Multi-view Tri-alignment for OCT to OCTA 3D Image Translation](https://mlsanity.com/paper/2504.01428/) (2025)
- [WFM: 3D Wavelet Flow Matching for Ultrafast Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2604.21146/) (2026)
- [Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation](https://mlsanity.com/paper/2411.06106/) (2024)
- [MRGen: Segmentation Data Engine for Underrepresented MRI Modalities](https://mlsanity.com/paper/2412.04106/) (2024)
- [ContourDiff: Unpaired Medical Image Translation with Structural Consistency](https://mlsanity.com/paper/2403.10786/) (2024)
- [Diff-DTI: Fast Diffusion Tensor Imaging Using A Feature-Enhanced Joint Diffusion Model](https://mlsanity.com/paper/2405.15830/) (2024)
- [Similarity-aware Syncretic Latent Diffusion Model for Medical Image Translation with Representation Learning](https://mlsanity.com/paper/2406.13977/) (2024)
- [Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2408.13733/) (2024)
- [ImmunoDiff: A Diffusion Model for Immunotherapy Response Prediction in Lung Cancer](https://mlsanity.com/paper/2505.23675/) (2025)
- [Pre to Post-Treatment Glioblastoma MRI Prediction using a Latent Diffusion Model](https://mlsanity.com/paper/2510.17851/) (2025)
- [Scalable Diffusion Transformer for Conditional 4D fMRI Synthesis](https://mlsanity.com/paper/2511.22870/) (2025)
- [Multiscale Structure-Guided Latent Diffusion for Multimodal MRI Translation](https://mlsanity.com/paper/2603.12581/) (2026)
- [StyleX: A Trainable Metric for X-ray Style Distances](https://mlsanity.com/paper/2405.14718/) (2024)
- [Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2505.07866/) (2025)
- [Cascaded 3D Diffusion Models for Whole-body 3D 18-F FDG PET/CT synthesis from Demographics](https://mlsanity.com/paper/2505.22489/) (2025)
- [Privacy-Preserving Chest X-ray Classification in Latent Space with Homomorphically Encrypted Neural Inference](https://mlsanity.com/paper/2506.15258/) (2025)
- [Breast Cancer Detection in Thermographic Images via Diffusion-Based Augmentation and Nonlinear Feature Fusion](https://mlsanity.com/paper/2509.07277/) (2025)
- [Tumor Synthesis conditioned on Radiomics](https://mlsanity.com/paper/2509.24182/) (2025)
- [Lesion-Aware Post-Training of Latent Diffusion Models for Synthesizing Diffusion MRI from CT Perfusion](https://mlsanity.com/paper/2510.09056/) (2025)

## Abstract (arXiv)

Multi-modal images play a crucial role in comprehensive evaluations in medical image analysis providing complementary information for identifying clinically important biomarkers. However, in clinical practice, acquiring multiple modalities can be challenging due to reasons such as scan cost, limited scan time, and safety considerations. In this paper, we propose a model based on the latent diffusion model (LDM) that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping. The 3D LDM combined with conditioning using the target modality allows generating high-quality target modality in 3D overcoming the shortcoming of the missing out-of-slice information in 2D generation methods. The switchable block, noted as multiple switchable spatially adaptive normalization (MS-SPADE), dynamically transforms source latents to the desired style of the target latents to help with the diffusion process. The MS-SPADE block allows us to have one single model to tackle many translation tasks of one source modality to various targets removing the need for many translation models for different scenarios. Our model exhibited successful image synthesis across different source-target modality scenarios and surpassed other models in quantitative evaluations tested on multi-modal brain magnetic resonance imaging datasets of four different modalities and an independent IXI dataset. Our model demonstrated successful image synthesis across various modalities even allowing for one-to-many modality translations. Furthermore, it outperformed other one-to-one translation models in quantitative evaluations.