# Cross-Modality Neuroimage Synthesis: A Survey

arXiv: https://arxiv.org/abs/2202.06997 · HTML: https://mlsanity.com/paper/2202.06997/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 148 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 30 of 42)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation](https://mlsanity.com/paper/2002.02255/) (2020)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1810.06498/) (2018)
- [Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks](https://mlsanity.com/paper/1908.02498/) (2019)
- [Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2003.00273/) (2020)
- [Sparse Modeling for Image and Vision Processing](https://mlsanity.com/paper/1411.3230/) (2014)
- [Cross-modality image synthesis from unpaired data using CycleGAN: Effects of gradient consistency loss and training data size](https://mlsanity.com/paper/1803.06629/) (2018)
- [Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization](https://mlsanity.com/paper/1908.06965/) (2019)
- [Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis](https://mlsanity.com/paper/2008.03483/) (2020)
- [DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis](https://mlsanity.com/paper/1904.12894/) (2019)
- [mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis](https://mlsanity.com/paper/1909.11504/) (2019)
- [Simultaneous Super-Resolution and Cross-Modality Synthesis of 3D Medical Images using Weakly-Supervised Joint Convolutional Sparse Coding](https://mlsanity.com/paper/1705.02596/) (2017)
- [CollaGAN : Collaborative GAN for Missing Image Data Imputation](https://mlsanity.com/paper/1901.09764/) (2019)
- [An Adversarial Learning Approach to Medical Image Synthesis for Lesion Detection](https://mlsanity.com/paper/1810.10850/) (2018)
- [Multi-Domain Image Completion for Random Missing Input Data](https://mlsanity.com/paper/2007.05534/) (2020)
- [Self-Attentive Spatial Adaptive Normalization for Cross-Modality Domain Adaptation](https://mlsanity.com/paper/2103.03781/) (2021)
- [Semantic-Aware Generative Adversarial Nets for Unsupervised Domain Adaptation in Chest X-ray Segmentation](https://mlsanity.com/paper/1806.00600/) (2018)
- [FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesis](https://mlsanity.com/paper/2201.08953/) (2022)

## Built on by (most cited first, 7 of 7)

- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Medical Vision Generalist: Unifying Medical Imaging Tasks in Context](https://mlsanity.com/paper/2406.05565/) (2024)
- [Navigating Distribution Shifts in Medical Image Analysis: A Survey](https://mlsanity.com/paper/2411.05824/) (2024)
- [Cross-modality image synthesis from TOF-MRA to CTA using diffusion-based models](https://mlsanity.com/paper/2409.10089/) (2024)
- [fRegGAN with K-space Loss Regularization for Medical Image Translation](https://mlsanity.com/paper/2303.15938/) (2023)
- [ReMAP-PET: Beyond Visual Understanding -- Learning Region-Guided Metabolic Alignment Semantics from Brain PET](https://mlsanity.com/paper/2606.29577/) (2026)

## Abstract (arXiv)

Multi-modality imaging improves disease diagnosis and reveals distinct deviations in tissues with anatomical properties. The existence of completely aligned and paired multi-modality neuroimaging data has proved its effectiveness in brain research. However, collecting fully aligned and paired data is expensive or even impractical, since it faces many difficulties, including high cost, long acquisition time, image corruption, and privacy issues. An alternative solution is to explore unsupervised or weakly supervised learning methods to synthesize the absent neuroimaging data. In this paper, we provide a comprehensive review of cross-modality synthesis for neuroimages, from the perspectives of weakly supervised and unsupervised settings, loss functions, evaluation metrics, imaging modalities, datasets, and downstream applications based on synthesis. We begin by highlighting several opening challenges for cross-modality neuroimage synthesis. Then, we discuss representative architectures of cross-modality synthesis methods under different supervisions. This is followed by a stepwise in-depth analysis to evaluate how cross-modality neuroimage synthesis improves the performance of its downstream tasks. Finally, we summarize the existing research findings and point out future research directions. All resources are available at https://github.com/M-3LAB/awesome-multimodal-brain-image-systhesis