# Unified Multi-Modal Image Synthesis for Missing Modality Imputation

arXiv: https://arxiv.org/abs/2304.05340 · HTML: https://mlsanity.com/paper/2304.05340/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 61 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 27 of 27)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities](https://mlsanity.com/paper/2104.06231/) (2021)
- [One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation](https://mlsanity.com/paper/2204.13738/) (2022)
- [DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis](https://mlsanity.com/paper/1904.12894/) (2019)
- [mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis](https://mlsanity.com/paper/1909.11504/) (2019)
- [CollaGAN : Collaborative GAN for Missing Image Data Imputation](https://mlsanity.com/paper/1901.09764/) (2019)
- [Multi-Domain Image Completion for Random Missing Input Data](https://mlsanity.com/paper/2007.05534/) (2020)
- [One Model to Unite Them All: Personalized Federated Learning of Multi-Contrast MRI Synthesis](https://mlsanity.com/paper/2207.06509/) (2022)
- [A unified representation network for segmentation with missing modalities](https://mlsanity.com/paper/1908.06683/) (2019)
- [Generative Adversarial Training for MRA Image Synthesis Using Multi-Contrast MRI](https://mlsanity.com/paper/1804.04366/) (2018)
- [Undersampled MRI Reconstruction with Side Information-Guided Normalisation](https://mlsanity.com/paper/2203.03196/) (2022)
- [LungViT: Ensembling Cascade of Texture Sensitive Hierarchical Vision Transformers for Cross-Volume Chest CT Image-to-Image Translation](https://mlsanity.com/paper/2210.02625/) (2022)

## Built on by (most cited first, 30 of 40)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [Self-Consistent Recursive Diffusion Bridge for Medical Image Translation](https://mlsanity.com/paper/2405.06789/) (2024)
- [MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection](https://mlsanity.com/paper/2410.03010/) (2024)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [Regression is all you need for medical image translation](https://mlsanity.com/paper/2505.02048/) (2025)
- [Learning Modality-Aware Representations: Adaptive Group-wise Interaction Network for Multimodal MRI Synthesis](https://mlsanity.com/paper/2411.14684/) (2024)
- [Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised Code](https://mlsanity.com/paper/2501.18328/) (2025)
- [SAGCNet: Spatial-Aware Graph Completion Network for Missing Slice Imputation in Population CMR Imaging](https://mlsanity.com/paper/2508.07041/) (2025)
- [Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities](https://mlsanity.com/paper/2405.11828/) (2024)
- [FgC2F-UDiff: Frequency-guided and Coarse-to-fine Unified Diffusion Model for Multi-modality Missing MRI Synthesis](https://mlsanity.com/paper/2501.03526/) (2025)
- [Augmented Intelligence for Multimodal Virtual Biopsy in Breast Cancer Using Generative Artificial Intelligence](https://mlsanity.com/paper/2501.19176/) (2025)
- [Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning](https://mlsanity.com/paper/2503.01265/) (2025)
- [Multi-modal MRI Translation via Evidential Regression and Distribution Calibration](https://mlsanity.com/paper/2407.07372/) (2024)
- [DuDoUniNeXt: Dual-domain unified hybrid model for single and multi-contrast undersampled MRI reconstruction](https://mlsanity.com/paper/2403.05256/) (2024)
- [Deep infant brain segmentation from multi-contrast MRI](https://mlsanity.com/paper/2512.05114/) (2025)
- [ModalImmune: Immunity Driven Unlearning via Self Destructive Training](https://mlsanity.com/paper/2602.16197/) (2026)
- [Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention](https://mlsanity.com/paper/2605.06699/) (2026)
- [HyperMM : Robust Multimodal Learning with Varying-sized Inputs](https://mlsanity.com/paper/2407.20768/) (2024)
- [Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities](https://mlsanity.com/paper/2408.14585/) (2024)
- [Longitudinal Causal Image Synthesis](https://mlsanity.com/paper/2410.17691/) (2024)
- [Efficient Parameter Adaptation for Multi-Modal Medical Image Segmentation and Prognosis](https://mlsanity.com/paper/2504.13645/) (2025)
- [BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2506.03675/) (2025)
- [DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation](https://mlsanity.com/paper/2506.11691/) (2025)
- [F3-Net: Foundation Model for Full Abnormality Segmentation of Medical Images with Flexible Input Modality Requirement](https://mlsanity.com/paper/2507.08460/) (2025)
- [Pyramid Hierarchical Masked Diffusion Model for Imaging Synthesis](https://mlsanity.com/paper/2507.16579/) (2025)
- [CLARGA: Multimodal Graph Representation Learning over Arbitrary Sets of Modalities](https://mlsanity.com/paper/2512.11901/) (2025)
- [OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows](https://mlsanity.com/paper/2602.04144/) (2026)
- [Any-to-All MRI Synthesis: A Unified Foundation Model for Nasopharyngeal Carcinoma and Its Downstream Applications](https://mlsanity.com/paper/2602.08822/) (2026)
- [Towards Personalized Multi-Modal MRI Synthesis across Heterogeneous Datasets](https://mlsanity.com/paper/2602.19723/) (2026)

## Abstract (arXiv)

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.