# CNNs and GANs in MRI-based cross-modality medical image estimation

arXiv: https://arxiv.org/abs/2106.02198 · HTML: https://mlsanity.com/paper/2106.02198/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 91 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Deep MR to CT Synthesis using Unpaired Data](https://mlsanity.com/paper/1708.01155/) (2017)
- [SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1810.06498/) (2018)
- [Cross-modality image synthesis from unpaired data using CycleGAN: Effects of gradient consistency loss and training data size](https://mlsanity.com/paper/1803.06629/) (2018)
- [Domain-adversarial neural networks to address the appearance variability of histopathology images](https://mlsanity.com/paper/1707.06183/) (2017)
- [Deep CT to MR Synthesis using Paired and Unpaired Data](https://mlsanity.com/paper/1805.10790/) (2018)
- [Adversarial Synthesis Learning Enables Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1712.07695/) (2017)
- [Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection](https://mlsanity.com/paper/1802.07846/) (2018)
- [Synthesis of Positron Emission Tomography (PET) Images via Multi-channel Generative Adversarial Networks (GANs)](https://mlsanity.com/paper/1707.09747/) (2017)
- [Generative Adversarial Networks for Image-to-Image Translation on Multi-Contrast MR Images - A Comparison of CycleGAN and UNIT](https://mlsanity.com/paper/1806.07777/) (2018)
- [Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image](https://mlsanity.com/paper/1709.02073/) (2017)
- [Cross-modality (CT-MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets](https://mlsanity.com/paper/1901.11369/) (2019)
- [Learning Myelin Content in Multiple Sclerosis from Multimodal MRI through Adversarial Training](https://mlsanity.com/paper/1804.08039/) (2018)
- [Super-resolution PET imaging using convolutional neural networks](https://mlsanity.com/paper/1906.03645/) (2019)
- [Synthesizing CT from Ultrashort Echo-Time MR Images via Convolutional Neural Networks](https://mlsanity.com/paper/1807.10850/) (2018)

## Built on by (most cited first, 2 of 2)

- [Generative Adversarial Networks for Brain Images Synthesis: A Review](https://mlsanity.com/paper/2305.15421/) (2023)
- [MRI Cross-Modal Synthesis: A Comparative Study of Generative Models for T1-to-T2 Reconstruction](https://mlsanity.com/paper/2602.07068/) (2026)

## Abstract (arXiv)

Cross-modality image estimation involves the generation of images of one medical imaging modality from that of another modality. Convolutional neural networks (CNNs) have been shown to be useful in identifying, characterising and extracting image patterns. Generative adversarial networks (GANs) use CNNs as generators and estimated images are discriminated as true or false based on an additional network. CNNs and GANs within the image estimation framework may be considered more generally as deep learning approaches, since imaging data tends to be large, leading to a larger number of network weights. Almost all research in the CNN/GAN image estimation literature has involved the use of MRI data with the other modality primarily being PET or CT. This review provides an overview of the use of CNNs and GANs for MRI-based cross-modality medical image estimation. We outline the neural networks implemented, and detail network constructs employed for CNN and GAN image-to-image estimators. Motivations behind cross-modality image estimation are provided as well. GANs appear to provide better utility in cross-modality image estimation in comparison with CNNs, a finding drawn based on our analysis involving metrics comparing estimated and actual images. Our final remarks highlight key challenges faced by the cross-modality medical image estimation field, and suggestions for future research are outlined.